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Abstract 

ihis  thesis  was  based  on  the  hypothesis  that  Air  Forc.i 
Civil  Engineering  construction  managers  could  lessen  the 
impact  of  inexperience  through  the  use  of  lessons-learned. 
The  thesis  examined  the  potential  for  developing  an  on-line 
management  information  system  (MIS)  to  provide  better 
storage  and  retrieval  of  lessons-learned.  Emphasis  was 
placed  on  developing  a  WANG-based  system  that  would  be 
accessible  at  all  levels  of  Civil  Engineering  construction. 

Research  consisted  of  reviewing  MIS  development 
methodologies,  investigating  several  existing  information 
systems  -  both  general  and  lessons-learned  oriented  -  and 
determining  the  users’  requirements  for  the  proposed  MIS. 

The  objective  of  this  research  was  to  identify  factors 
and  procedures  that  should  be  considered  when  developing  a 
construction  management  oriented,  lessons-learned  management 
information  system  for  the  Civil  Engineering  WANG  computer. 
The  process  involved  reviewing  pertinent  literature,  logging 
onto  and  using  existing  on-line  information  systems  and 
interviewing  construction  managers  at  the  MAJCOM  and  AFRCEs. 

Conclusions  from  this  research  indicated  on-line 
information  systems  are  well  suited  to  lessons  learned.  All 
of  the  AFRCE  construction  managers  interviewed  stressed 
additional  use  of  lessons-learned  is  necessary.  Further, 


vii 


t  construction  managers  stated  the  proposed  system  should 
be  WANG-based,  menu-driven,  user-friendly  and  easily 
adaptable  to  the  changing  needs  of  the  user.  Menus  and 
system  characteristics  for  a  prototype  lessons-learned  MIS 
are  outlined  in  the  research.  The  author  recommended  a 
WANG-based  lessons-learned  MIS  prototype  be  developed,  using 
the  research  findings  as  a  starting  point,  with  further 
refinement  through  a  more  broad-based,  user  involvement. 
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A  MANAGEMENT  INFORMATION  SYSTEM 
FOR  CONSTRUCTION  MANAGEMENT 
LESSONS-LEARNED 


I .  Introduction 


General  Issue 

The  primary  mission  of  United  States  Air  Force  (USAF) 
Civil  Engineering  (CE),  as  outlined  in  Air  Force  Regulation 
85-10,  Operations  and  Maintenance  of  Real  Property,  includes 
the  acquisition,  construction,  maintenance  and  operation  of 
real  property  facilities  (AFR  85-10:  2).  With  an  annual 
construction  budget  approaching  a  billion  dollars,  the 
construction  portion  of  this  mission  is  significant 
(Majdanik,  1989). 

To  meet  mission  requirements,  Civil  Engineering  uses  a 
combination  of  in-house  and  contract  work  forces  (AFR  85-1: 
88).  As  reductions  in  manpower  shrink  the  size  of  in-house 
work  forces,  Civil  Engineering  must  increasingly  rely  on 
construction  contracts  to  accomplish  real  property 
maintenance,  repair  and  modification.  Unfortunately, 
frequently  the  military  personnel  who  manage  construction 
are  young  and  do  not  have  the  requisite  experience. 

A  national  field  survey  of  all  principle  parties  in 
construction  contracts  (owners,  contractors,  construction 
managers,  clients,  etc.),  including  the  U.S.  Army  Corps  of 
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Engineers,  the  Naval  Facilities  Engineering  Command  and  the 
Air  Force,  found  that  up  to  76  percent  of  the  owners  and 
contractors  felt  that  inspectors  did  not  have  the  necessary 
experience  to  inspect  construction  work  (Task  CoaBittee  on 
Inspection,  1972:  219-234). 

Within  Air  Force  Civil  Engineering,  the  problem  of 
inexperienced  inspectors  is  even  more  pervasive.  Williams 
surveyed  design  engineers,  contract  administrators,  contract 
management  chiefs  and  inspectors  to  evaluate  their 
perceptions  of  Air  Force  inspectors’  job  performance.  The 
research  fo?md  "significant  reservations"  about  the 
capabilities  and  experience  of  inspectors  across  almost  all 
respondents.  Williams  verified  that  within  Civil 
Engineering,  a  "lack  of  inspector  experience  and  training 
continually  surfaced  as  a  problem”  (Williams,  1986:  1-5, 

72)  . 

OSAF  construction  managers  at  all  levels  —  base,  major 
command  (MAJCOM),  Air  Force  Regional  Civil  Engineer  ( AFRCE ) , 
and  Air  Staff  --  document  negative  lessons-learned,  or 
common  pitfalls,  in  construction  management  and  ways  to 
avoid  them  (Bradshaw,  1988).  Similarly,  throughout  the  Air 
Force,  individual  bases  and  personnel  have  found  positive 
lessons-learned,  i.e.  construction  methods,  materials  and 
techniques,  that  work  extremely  well.  Without  a  clear 
method  of  consolidation  and  dissemination  of  this  critical 
information,  however,  construction  managers  frequently 
repeat  the  same  errors  instead  of  learning  from  past 
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mistakes  of  others.  Likewise,  when  a  particularly 
successful  method  or  material  is  found,  the  benefit  of  that 
discovery  is  not  shared  with  other  inspectors  in  the  Air 
Force.  Experience,  or  the  ability  to  recall  and  use  past 
lessons -learned,  greatly  reduces  the  recurrence  of  common 
mistakes  and  increases  the  use  of  successful  techniques. 

Specific  Problem 

Air  Force  Civil  Engineering  construction  contract 
inspectors  are  inexperienced  and  frequently  fail  to  benefit 
from  the  experiences  of  others.  A  methodology  is  needed  to 
develop  an  on-line  computer-based  management  information 
system  containing  construction  management  lessons-learned  to 
improve  information  crossfeed. 

Scope  and  Limitations 

The  research  scope  was  limited  to  determining  the 
requirements  necessary  to  develop  a  WANG  based  lessons- 
learned  MIS  from  an  end-user’s  perspective.  A  WANG  based 
system  was  selected  as  the  WANG  computer  is  common  to 
virtually  all  CE  organizations.  The  specific  area  of 
research  focused  on  construction  managers  of  large 
construction  projects  managed  by  the  AFRCEs,  either 
independently  or  with  the  assistance  of  the  Army  Corps  of 
Engineers  or  the  Naval  Facilities  Command.  Construction 
managers  of  large  scale  projects  were  selected  as  the  "end- 
users"  because  they  typically  have  the  most  experience  of 
all  CE  construction  managers. 
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Objective 

The  research  objective  was  to  identify  factors  and 
procedures  that  should  be  considered  when  developing  a 
construction  management  oriented  "lessons-learned'' 
management  information  system  for  the  Civil  Engineering  WANG 
computer . 


Research  Questions 

The  following  research  questions  were  used  to  reach  the 

objective  stated  above: 

1 .  Bow  are  generic  management  information  systems 
developed? 

2.  Are  there  on-line  "lessons-learned"  management 
information  systems  in  use?  If  so,  how  have  these 
systems  been  developed? 

3.  What  common  factors  made  the  existing  systems 
successful  or  less  than  successful? 

4.  How  should  a  WANG-based  lessons- learned  system  be 
developed  to  meet  the  needs  of  AF  construction 
managers? 

5.  From  the  user’s  perspective,  how  should  the  WANG 
lessons-learned  MIS  be  structured? 


Justif ication 

Civil  Engineering  managers  from  Air  Staff  to  base  leve1 
recognize  lessons-learned  feedback  and  crossfeed  can  yield 
substantial  benefits  and,  to  a  great  extent,  buffer  the 
impact  of  inexperience .  HQ  DSAF  Engineering  and  Services 
Installation  Development  Division  (HQ  OSAF/LEEDP)  initiated 
an  investigational  engineering  program  project  specifically 
for  the  development  of  computer  software  for  facility 
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acquisition  lessons-learned  (Schmidt,  1989).  Proponents  of 

this  project  are  convinced, 

Lessons-learned  in  Air  Force  construction  are  not 
effectively  utilized.  The  lack  of  a  central 
databank  of  information  on  facilities  acquisitions 
lessons-learned  allows  our  component  activities  to 
repeat  mistakes  instead  of  learning  from  others. 
(Bradshaw,  1988) 

The  Air  Force  Engineering  and  Services  Center 

(AFESC)  at  Tyndall  AFB  has  initiated  a  "Reliability  and 

Maintainability  Lessons-learned  Program  to  enhance  the 

quality  and  performance  of  facilities  and  supporting 

systems"  designed,  built  and  maintained  within  the  Air  Force 

by  Civil  Engineering  (Wentland,  1989).  To  support  the 

Reliability  and  Maintainability  Lessons-learned  Program, 

AFESC  is  currently  working  to  ease  and  improve  the  flow  of 

information  among  CE  units  throughout  the  Air  Force  by 

upgrading  the  telecommunication  capabilities  of  the  Work 

Information  Management  System  (WIMS)  installed  at  119  CE 

locations  world -wide  (Wentland,  1989). 

All  of  the  Air  Force  Regional  Civil  Engineers  (AFRCEs), 

contacted  in  the  U.S.  expressed  concerns  about  the  need  to 

make  better  use  of  lessons-learned.  Colonel  Ralph  Hodge,  at 

the  Western  Division  AFRCE,  said 

Using  past  experiences  to  improve  future 
performance  is  crucial  if  we  want  to  stay  in  the 
construction  business .  A  prototype  hospital 
project  recently  completed  here  had  hundreds  of 
good,  solid  lessons  on  how  to  build  a  hospital. 

All  we  lack  is  a  way  to  collect  and  share  those 
lessons.  (Hodge,  1989) 
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Gary  Lynne,  at  the  Central  Division  AFRCE  in  Dallas,  related 
how  the  Central  Division  AFRCE  continually  strives  to  learn 
from  past  experiences,  including  the  recent  gathering  of 
nine  architectural  firms  who  have  worked  with  AFRCE  Central 
together  to 

...talk  about  all  the  things  we  have  done  right, 
and  all  the  things  we  are  doing  wrong.  If  we 
don’t  take  the  time  to  learn  from  previous  lessons 
we’ll  repeat  our  failures  and  miss  out  on  a  lot  of 
potential  successes.  (Lynne,  1989) 

Lynne  acknowledged  this  as  only  one  approach  to  learning 

from  past  experiences  and  stressed  much  more  can  and  should 

be  done  in  the  area  of  lessons-learned.  According  to 

Lynne, 

Quite  frankly,  we  aren’t  making  as  much  use  from 
our  past  experiences  as  we  should.  There  is 
unfortunately  no  real,  formal  feedback  system 
available. . .  but  one  is  definitely  needed.  A 
computerized  system  on  the  WANG  is  an  option  that 
should  be  explored.  (Lynne,  1989) 

Charles  Smith,  at  the  Eastern  Division  AFRCE  in  Atlanta, 

echoed  the  sentiments  of  his  counterparts  as  he  explained, 

Internal  to  each  AFRCE,  the  Branch  Chiefs  and 
Project  Managers  are  responsible  for  crossfeeding 
lessons -learned.  In-house  this  is  done  fairly 
well,  but  it’s  still  less  than  perfect.  We  try  to 
make  maximum  use  of  crossfeed.  Externally, 
crossfeed  between  the  various  AFRCEs  is  even  less 
efficient.  Right  now  there  is  no  official 
feedback  system ...  there  should  be,  but  there 
isn’t.  (Smith,  1989) 

Gary  Erickson,  at  the  Strategic  Air  Command  AFRCE  at  Offutt 
AFB,  has  made  tremendous  use  of  lessons-learned  to  improve 
SAC’s  facility  acquisition  program.  AFRCE  SAC  employs  both 
an  automated  and  manual  approach  to  lessons-learned. 
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AFRCE  SAC  developed  a  WANG-based  lessons- learned  program  "to 
combine  the  lessons -learned  from  over  25  SAC  bases  and  get  a 
look  at  what’s  going  on  across  the  command"  (Erickson, 

1989).  A  hard  copy  booklet  of  lessons-learned  has  also  been 
written  regarding  the  B-1B  program  recently  completed  by 
AFRCE  SAC.  This  booklet  is  for  the  bases’  and  Army  Corps  of 
Engineers  (COE)  district  engineers’  use  (Erickson,  1989). 

The  AFRCE  SAC  lessons-learned  WANG  program  is  still  in  the 
development  stages. 

Although  monetary  savings  yielded  through  the  use  of 
lessons-learned  are  at  best  only  an  estimation,  an  analysis 
based  purely  on  historical  data  provides  at  least  a 
reasonable  assessment  of  the  rough  order  of  magnitude  of 
potential  savings.  For  example,  it  is  an  accepted  reality 
that  virtually  all  facility  construction  contracts  are 
modified  or  changed  during  the  construction  process.  In 
fact,  for  planning  purposes,  the  cost  of  most  government 
construction  projects  can  be  expected  to  increase  "25  to  30 
percent  over  the  life  of  the  project,  from  inception  to 
completion"  (Majdanik,  1989).  Using  these  percentages,  over 
$200  million  of  the  Air  Force  $868  million  FY90  Military 
Construction  Program  (MCP)  budget  may  be  due  to  project 
changes.  If  only  a  small  percentage  of  those  project 
changes  costs  could  be  avoided  by  the  application  of 
lessons-learned,  monetary  savings  would  be  substantial. 
Tucker  concurs  with  this  assessment  and  notes  that  due  to 
widely  varying  facility  construction  costs,  scope  and 
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complexity,  "Lessons -learned. . .  cannot  be  quantified  into 
savings  of  time  and  money.  Yet  large  savings  of  time  and 
money  are  possible."  (Tucker,  1984:  1). 

Nonmonetary  costs  of  not  using  lessons-learned  are 
equally  difficult  to  quantify.  However,  the  results  of 
failing  to  benefit  from  past  experiences  are  apparent  in 
many  areas.  For  example,  Keller  researched  the  causes  of 
functional  deficiencies  in  tactical  aircraft  maintenance 
facilities  and  discovered  that  a  primary  cause  of  the 
functional  deficiencies  was  “most  maintenance  facilities  are 
designed  as  one-of-a-kind  without  the  benefit  of  lessons- 
learned  from  construction  of  similar  type  facilities" 
(Keller,  1987:  59).  The  research  concluded  failure  to 
benefit  from  lessons-learned  created  a  "negative  impact  on 
the  efficiency  of  maintenance  operations  and  the  number  of 
man-hours  normally  required  to  perform  various  tasks" 
(Keller,  1987:  59). 

While  Keller  considered  only  tactical  aircraft 
maintenance  facilities,  the  USAF  builds  several  other 
standard  facilities  such  as  housing,  administrative,  child 
care,  dormitories,  gymnasiums,  etc.  every  year.  Each  of 
these  facilities  is  built  with  the  same  methodology  as  the 
maintenance  facilities  addressed  by  Keller,  without  the 
benefits  of  lessons-learned.  When  the  "negative  impact  on 
the  efficiency"  of  operations  noted  by  Keller  in  maintenance 
facilities  is  multiplied  across  all  facilities,  the 
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nonmonetary  impact  of  failing  to  use  lessons-learned  in 
facility  acquisition  also  becomes  substantial. 

Background 

Knowledge  of  two  distinctly  different  subjects  is 
needed  to  understand  the  research  problem.  The  subject 
areas  in  question  are  Air  Force  construction  contracting  and 
management  information  systems.  This  section  provides 
knowledge  necessary  to  understand  why  experience  is  crucial 
to  construction  management .  Further,  this  section  explores 
how  an  MIS  can  provide  "experience"  to  the  unexperienced. 

First,  this  section  provides  a  brief  overview  of  the 
Air  Force  construction  contract  process,  leading  to  an  in- 
depth  look  at  the  sub -area  of  construction  management.  The 
construction  management  sub-area  details  the  primary 
responsibilities  and  necessary  qualifications  of 
construction  managers  or  inspectors.  A  principle 
qualification  of  construction  managers  is  experience. 

Second,  this  section  provides  am  overview  of  1)  what  a 
management  information  system  (MIS)  is  and,  2)  how  an  MIS 
may  be  used  to  collate,  organize,  store  and  retrieve  the 
collective  experiences  or  "lessons-learned"  of  Air  Force 
construction  managers. 

Air  Force  Construction  Contract  Process.  Merrill  and 
Torchia  provide  am  excellent  overview  of  the  Air  Force 
construction  contract  process,  sufficiently  simplified  for 
this  research,  yet  still  accurate.  A  construction  contract 
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is  a  legal  agreement  between  two  parties,  the  owner  and  the 
contractor.  The  contractor  is  hired  to  perform  construction 
for  the  owner.  A  construction  contract  is  generally  used 
when  a  valid  work  request  exceeds  the  capabilities  of  OSAF 
Civil  Engineering.  The  principle  parties  in  the 
construction  contract  process  are  the  user  (originator  of 
the  work  requirement),  the  contracting  officer,  the 
designer,  the  construction  manager  (on  relatively  small  jobs 
this  is  also  the  inspector),  and  the  contractor.  For 
purposes  of  control,  authority  to  direct  the  contractor  is 
limited  to  the  contracting  officer.  However,  within  the 
scope  of  the  contract,  the  construction  manager  has  indirect 
control  over  the  contractor.  The  user  and  designer  are 
involved  in  the  contractual  process  in  a  third  party  role, 
with  their  interests  being  brought  to  the  contractor  by  the 
construction  manager  (Merrill  and  Torchia,  1982:  5-8). 

These  relationships  are  shown  in  Figure  1. 

Typically,  the  construction  manager  for  small 
maintenance  and  repair  construction  efforts  falls  into  one 
of  three  groups  -  a)  lieutenant  or  young  captain,  b)  mid  to 
low-range  noncommissioned  officer  in  the  E-4  to  E-6  grade 
levels,  or  c)  civilian  in  the  GS-7  to  GS-10  grade  levels. 

Of  these  groups,  the  military  construction  managers  tend  to 
be  the  least  experienced,  primarily  due  to  frequent  rotation 
among  jobs  and  bases  (Meister,  1989). 
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Figure  1 .  Contract  Relationships 
(Merrill  and  Torchia,  1982:  5-8) 


MCP  Projects.  One  variation  on  the  description  of 
the  Air  Force  construction  process  provided  by  Merrill  and 
Torchia  concerns  large  scale  construction  projects  funded 
and  built  under  the  Military  Construction  Program  (MCP). 

The  design  and  construction  management  of  large  scale  MCP 
projects  are  accomplished  for  the  Air  Force  by  either  the 
Army  Corps  of  Engineers  (COE)  or  Naval  Facilities 
Engineering  Command  (NAVFAC).  Although  the  COE  and/or 
NA7FAC  perform  as  the  design  and  construction  agent  for  the 
Air  Force,  the  Air  Force  provides  a  project  manager  for  each 
MCP  project  interface  between  the  Air  Force  and  the  COE  or 
NAVFAC  (AFR  86-1,  1984:  1-20).  The  relationships  of  the 
key  players  in  the  MCP  contractual  process  are  shown  in 
Figure  2  on  the  following  page. 
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Figure  2.  MCP  Cont.ract.ual  Relationships 
(Beally ,  1989) 


Within  the  MCP  arena,  the  AF  project  manager  i3  in  a 
unique  position.  Charged  with  the  responsibility  for 
ensuring  the  COE/NAVFAC  provide  the  facility  being  paid  for 
by  the  Air  Force,  the  AF  project  manager  often  feels 
frustrated  by  his  lack  of  control.  Examples  of  difficulties 
experienced  by  Air  Force  project  managers  on  MCP  projects 
are  plentiful.  AF  project  managers  on  MCP  projects  are  not 
allowed  to  participate  in  contract  negotiations.  Frequently 
results  of  negotiations  are  not  provided  in  a  timely  manner 
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to  the  USAF.  Secrecy  on  the  part  of  the  COE  often  adds  to 
the  A F  project  manager’s  suspicions  that  the  COE  finds  it 
easier  to  request  more  money  from  the  Air  Force  than  to 
oppose  a  contractor’s  claims.  The  relationships  between  the 
AF  project  manager  and  the  COE/NAVFAC  construction 
management  staff  are  often  strained  by  the  bureaucracy  of 
the  two  separate  services  (Tucker,  1984:  47-50). 

The  information  above  is  provided  not  as  an  indictment 
of  COE  or  NAVFAC;  on  the  whole,  they  can  and  do  provide  an 
outstanding  service  to  the  Air  Force  in  construction 
management.  This  information  is  offered  instead  to 
demonstrate  that  under  the  MCP  process,  AF  construction 
managers  must  rely  even  more  heavily  on  experience  than 
their  non-MCP  counterparts.  Keller  noted  that  the  COE  does 
"not  use  a  lessons-learned  system  to  improve  the  designs  of 
like  facilities"  (Keller,  1987:  56).  Equally  disturbing, 
Tucker  stressed  that  although  the  experience  should  match 
the  job,  frequently, 

This  isn’t  easy  to  accomplish  because  the  people 
with  needed  expertise  have  good  permanent  jobs  and 
they  are  not  anxious  to  leave  them  for  a  two  or 
three  year  construction  project.  . . .  The  COE  has  a 
problem  with  lower  grades  in  its  field  offices. 
Traditionally,  Corps  of  Engineers  district  staffs 
hold  higher  government  service  grades  than  its 
field  staffs.  Thus,  the  expertise  gravitates  to 
the  stable  district  staff  while  field  offices 
generally  are  undergraded  and  understaffed. 

(Tucker,  1984:  50) 

Construction  Hanagp.mfint .  Construction  management 
consists  of  controlling  administrative,  legal,  financial  and 
behavioral  elements  of  the  construction  process  (Levitt, 
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1987:  86).  Within  the  Air  Force,  construction  is  governed 
by  Air  Force  Regulation  89-1  (AFR  89-1),  Design  and 
Construction  Management.  AFR  89-1  outlines  the 
responsibilities,  policies  and  procedures  necessary  for  the 
construction  of  Air  Force  facilities  (AFR  89-1:1-1). 
According  to  AFR  89-1,  the  main  purpose  of  construction 
management/inspection  is  to  ensure  the  contractor  adheres  to 
the  approved  drawings  and  specifications  (AFR  89-1:  13-1). 
The  key  responsibilities  of  the  construction  manager  as 
outlined  in  AFR  89-1  follow. 

The  inspector  must  have  an  understanding  of 
construction  practices  that  will  allow  recognition  of 
improper  construction.  The  inspector  must  possess  a 
thorough  knowledge  of  pertinent  contracting  regulations  to 
evaluate  whether  or  not  the  contractor  is  in  compliance  with 
the  specifications.  Although  the  inspector  does  not  have 
contractual  authority  to  direct  the  contractor,  the 
inspector  is  the  technical  representative  of  the  contracting 
officer.  In  this  capacity,  it  is  the  construction  manager 
who  must  recognize  a  problem  exists,  initiate  a  conference 
between  the  contracting  officer,  the  construction  manager 
and  the  contractor  and  "assist  in  the  interpretation  of  the 
technical  provisions  and  contractual  documents’’  to  resolve 
the  problem  (AFR  89-1:  13-1  to  13-4). 

The  recurrent  theme  in  these  AF  construction  manager 
responsibilities  is  knowledge.  As  a  profession, 
construction  management  relies  on  knowledge  based  on 
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experience,  rather  than  knowledge  based  on  education 
(Levitt,  1987:  88).  Many  projects  have  proven  the 
availability  of  experience  is  directly  related  to 
construction  project  success  (Ashley,  1987:  74). 
Unfortunately,  as  stated  earlier,  CE  inspectors  lack  a 
significant  amount  of  experience  in  construction  management. 

A  structured  approach  is  needed  to  capture  the  limited 
individual  experiences  of  the  many  construction  managers  in 
the  Air  Force.  Once  captured  and  organized  this  pool  of 
experience  in  the  form  of  lessons- learned  can  then  be  made 
available  to  all  CE  construction  managers.  There  are 
various  methods  or  approaches  suitable  to  solving  this 
problem,  which  when  reduced  to  its  most  basic  level,  is 
simply  one  of  information  sharing.  Management  information 
systems  are  well  suited  to  the  task  of  information  sharing, 
and  were  therefore  explored. 

Management  Information  Systems.  "Management 
information  system"  is  a  common  term  that  has  been  applied 
to  a  variety  of  computers,  software  programs  and  other 
assorted  information  tools  used  by  today’s  managers.  Lucas 
defines  an  information  system  as  ”a  set  of  organized 
procedures  that,  when  executed,  provides  information  to 
support  decision  making  and  control  of  the  organization” 
(Lucas,  1986:  10).  Sprague  noted  that  management 
informati">n  systems  were  the  first  attempts  of  information 
systems  professionals  to  provide  managers  with  the 
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information  that  is  critical  to  effective  and  efficient  job 
performance  (Sprague,  1980:  10-26). 

Management  information  systems  can  be  either  manual  or 
computer  based.  According  to  Lucas,  management  information 
systems  have  been  around  much  longer  than  the  term  MIS, 
coined  in  the  late  1960s. 

Since  people  first  inhabited  the  earth,  there  have 
been  information  systems.  Individuals, 
organizations,  and  nations  have  always  collected 
and  processed  "intelligence".  Early  information 
systems  were  highly  informal  and  involved  the 
exchange  of  news,  stories,  and  anecdotes  with 
neighbors.  (Lucas,  1986:11) 

As  noted  by  Sprague,  management  information  systems 
increase  both  the  effectiveness  and  efficiency  of  managers 
by  providing  information.  Viewed  from  a  slightly  different 
perspective,  management  information  systems  are  essentially 
a  vehicle  for  information  sharing.  In  the  case  of  Air  Force 
construction  managers  trying  to  overcome  limited  experience, 
the  information  needed  to  be  shared  consists  of  the  positive 
and  negative  lessons-leamed  by  their  predecessors  and 
counterparts . 

Information  sharing  has  made  tremendous  gains  with  the 

widespread  use  of  computers.  Senn  notes. 

Computer  based  information  systems  make  possible 
the  smooth  and  efficient  operations  of  airline 
reservations  offices,  hospital  records 
departments,  accounting  and  payroll  functions, 
electronic  banking,  telephone  switching  systems 
and  countless  other  applications,  both  large  and 
small.  (Senn,  1984:4) 

There  are  a  plethora  of  computer  based  information 
sharing  systems  on-line  and  available  for  use  by  the  public 
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and  commercial  sectors.  The  diversity  of  the  information 
available  can  be  seen  in  the  following  few  examples:  Defense 
Technical  Information  Center  (DTIC),  NOAA  National 
Meteorological  Database,  Biology  Network,  Lunar  and 
Planetary  Institute  Library  Information  Center,  Automated 
Case  Information  Management  System,  Antitrust  Management 
Information  System  (Zarozny,  1987).  Other  on-line  systems 
such  as  COMPUSERVE  offer  a  single  point  source  of 
information  on  various  subjects  from  the  stock  market  and 
real  estate  investments  to  electronic  travel  services  and 
shopping  via  computer  (Online  Today,  1989:  1-20). 

Air  Force  Civil  Engineering  recognized  the  importance 
of  computers  in  the  management  of  information  in  the  early 
1980s.  As  a  result,  the  WANG-based  Work  Information 
Management  System  (WIMS)  was  developed  (Wentland,  1989).  As 
stated  in  the  BCE’s  Guide  to  WIMS,  "Information  is  power. 
Through  visibility  of  information. . .WIMS  improves  customer 
service,  enhances  control  of  resources  and  allows 
information  sharing."  (BCE’s  Guide,  1988:1).  Although  not 
currently  utilized  in  a  construction  management  lessons- 
learaed  capacity,  "the  WANG  is  fully  capable  of  handling 
such  a  task  if  the  user’s  requirements  were  defined  in  this 
area"  (Wentland,  1989). 

One  of  the  most  significant  aspects  of  the  success  or 
failure  of  an  MIS  involves  the  human  resource.  According  to 
Eldin  Garrison,  "It  is  difficult,  in  fact,  t.o  list  any 


cause  of  failure  that  does  not  have  its  origins  in  some  kind 
of  human  interface  with  the  machine.”  (Garrison,  1980:  15). 

From  a  Civil  Engineering  viewpoint,  a  construction 
management  lessons-leamed  MIS  must  meet  the  needs  of  the 
construction  manager.  For  a  Civil  Engineering  construction 
manager,  the  best  user  interface  for  a  lessons- learned 
management  information  system  would  be  a  computer  system 
common  to  all  CE  units.  This  would  have  to  be  the  WANG- 
based  minicomputer  system,  currently  integrated  in  almost 
all  Civil  Engineering  squadrons  Air  Force  wide  (BCE’s  Guide, 
1988:  1-23). 

Summary 

Civil  Engineering  construction  managers  are  frequently 
young  and  inexperienced .  As  a  profession,  construction 
management  relies  heavily  on  experience  to  successfully 
complete  large  and  complex  construction  projects.  One  way 
AF  construction  managers  can  expand  their  experience  base  is 
through  the  use  of  lessons- learned,  i.e.  the  positive  and 
negative  experiences  of  others.  The  AFRCEs,  responsible  for 
the  management  of  large  scale  MCP  construction,  all  agree 
better  use  of  lessons-learned  offers  tremendous  benefits  in 
AF  construction  management.  Although  many  lessons-learned 
are  documented,  they  are  usually  accessible  only  to  the 
original  authors,  or  are  filed  away  never  to  be  seen  again. 
Management  information  systems  are  ideally  suited  to  this 
type  of  information  sharing  problem.  This  research 
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establishes  a  methodology  for  the  development  of  an  on-line, 
WANG  computer  based  lessons-learned  management  information 
system  for  the  use  of  AJ?  construction  managers;  and,  the 
characteristics  of  what  a  user-developed  prototype  would  be. 

Chapter  II  presents  a  detailed  look  at  past  research  in 
the  area  of  management  information  systems  and  lessons- 
learned  databases.  Chapter  III  provides  the  methodology 
used  in  this  study  to  solve  the  research  problem.  Chapters 
IV  and  V  present  the  research  findings  and  Chapter  VI  draws 
final  conclusions  and  offers  recommendations  for  further 
research. 
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Meeting  "the  research  objectives  required  collection  of 


primary  data.  Although  the  literature  provided  a  firm 
understanding  of  the  important  factors  and  techniques  of  MIS 
development,  it  lacked  substance  in  three  major  areas:  a) 

How  do  other  lessons-learned  MIS’s  operate?  b)  How  should  a 
lessons- learned  MIS  be  implemented  on  the  Air  Force  CE  WANG 
mini-computer?  and ,  c)  How  would  the  users  of  the  proposed 
MIS  structure  the  system?  This  void  in  the  research 
knowledge  was  filled  through  the  use  of  surveys  and  direct 
observation .  Direct  observation  was  very  helpful  in 
learning  how  other  MlS’ s  operate.  Surveys  provided  the 
answers  to  how  an  MIS  is  implemented  on  the  WANG  and  how  the 
users  would  structure  the  proposed  MIS. 

As  outlined  in  William  C.  Emory’s  Business  Research 
Methods .  surveys  can  be  either  interviews  or  questionnaires 
depending  upon  the  survey  strategy  (Emory,  1985:  202).  The 
primary  data  for  this  research  was  collected  via  a 
combination  of  personal  interviews,  telephone  interviews  and 
electronic  mail.  These  methods  were  used  in  lieu  of  written 
questionnaires  based  on  the  need  to  approach  the  data 
gathering  process  in  an  exploratory  fashion.  Additionally, 
personal  interviews,  or  telephone  interviews  offer  the 
researcher  much  more  latitude  when  dealing  with  exploratory 
research  of  a  complex  subject  (Emory,  1985:  203). 
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Population  and  Sample 

The  population  of  interest  for  this  research  consisted 
of  all  Air  Force  major  construction  program  Managers,  and 
all  lessons-leamed  management  information  systems. 

However,  since  no  statistical  inferences  were  to  be  made  on 
the  data,  and  there  was  no  need  to  generalize  to  a 
population  parameter,  a  nonprobability  purposive  judgement 
sample  was  considered  adequate  for  the  exploratory  research 
desired.  Consequently ,  a  sample  consisting  only  of  the 
construction  program  managers  at  major  AFRCEs  was  deemed 
sufficient  to  establish  the  user's  requirements.  Similarly, 
a  sample  consisting  of  major  DOD  lessons- learned  systems  was 
considered  adequate.  Data  for  this  research  was  collected 
from  HQ  OSAF  Installation  Development  Division  (LEED) , 
Washington  D.C.;  the  Air  Force  Engineering  and  Services 
Center  (HQ  AFESC),  Tyndall  AFB,  FL;  HQ  AFLC  Systems 
Management  Office,  Wright  Patterson  AFB,  OH;  HQ  AFLC 
Directorate  of  Engineering  (HQ  AFLC/DER) ,  Wright  Patterson 
AFB,  OH;  the  Eastern,  Western,  Central  and  SAC  Air  Force 
Regional  Civil  Engineers  (AFRCEs)  and  the  Naval  Air  Test 
Center,  Rotary  Wing  Division,  Patuxent  River  Naval  Air 
Station,  MD. 

The  data  obtained  are  not  intended  to  be  statistical, 
nor  are  the  data  to  be  taken  as  representative  of  Air  Force 
wide  construction  program  managers’  requirements.  A  cross 
sectional  study  is  adequate  for  the  needs  of  the  research, 
because  it  provides  a  simple  snapshot  in  time  of  the  users’ 
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requirements  for  the  proposed  MIS.  The  designers  and 
operators  of  every  MIS  researched  stressed  that  a  lessons- 
learaed  MIS  is  a  "very  dynamic ,  continually  evolving" 
entity,  which  must  have  sufficient  flexibility  to  adapt  to 
the  changing  needs  of  the  user  (Grimsley  and  others,  1989). 
Based  on  this  consensus ,  the  data  provided  by  this  snapshot 
in  time  is  adequate  to  initiate  a  lessons-  learned  MIS 
prototype,  with  the  full  expectation  that  the  prototype 
will,  by  design,  be  changed  and  modified  by  the  users. 

The  qualitative  nature  of  the  selected  methodology  was 
driven  by  the  goal  of  the  research  —  to  identify  the 
subjective  user’s  needs  —  to  aide  in  the  development  of  a 
lessons -learned  MIS.  Quantitative  research  was  ruled  out 
because  the  opinions,  ideas  and  free-thinking  concepts 
desired  are  not  quantitative  data  (Davis,  1988). 

Collgctjaa  qf  Rat,gi 

Preliminary  interview  questions  were  developed  based  on 
information  gained  in  the  literature  review  and  direct 
observation  of  several  on-line  systems.  On-line  systems 
observed  included  the  Air  Force  Lessons-Leamed  Database, 
Naval  Aviation  Lessons-Learned  Database,  Reliability  and 
Maintainability  Checklist  System,  and  other  general 
information  sharing  systems  such  as  the  Lunar  and  Planetary 
Institute  System,  NOAA  National  Meteorological  Database 
System  and  the  BIONET  System. 
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The  majority  of  the  questions  were  intentionally  open 
ended  to  allow  the  respondents  the  opportunity  to  bring  out 
additional  information.  Although  the  preliminary  questions 
were  reviewed  by  AFIT  staff  to  ensure  face  validity  and 
reliability,  as  was  anticipated,  several  respondents 
surfaced  additional  facets  of  the  WANG  and  MIS  questions 
that  were  beneficial  to  the  research.  For  this  type  of 
exploratory  research,  open,  semi -structured  questions  are 
highly  recommended  (Emory,  1985:  203). 

An  additional  motivation  to  use  interviews  in  lieu  of 
written  questionnaires  was  the  geographic  closeness  of 
several  sources.  HQ  AFLC/DER,  in  charge  of  several  large- 
scale  construction  projects,  offered  much  insight  from  a 
user’ s  perspective  of  the  proposed  management  information 
system.  HQ  AFALC/LSL  provided  significant  insight  into  the 
concept  of  USAF  lessons -learned  databases.  All  of  the 
organizations  noted  are  located  on  Wright  Patterson  AFB, 
collocated  with  AFIT  and  the  researcher. 

Due  to  limited  travel  funds,  HQ  Air  Force  Engineering 
and  Services  Center  (HQ  AFESC)  at  Tyndall  AFB,  and  the  Naval 
Air  Test  Center,  Rotary  Wing  Division  (NATC)  at  the  Patuxent 
River  Naval  Air  Station  were  selected  as  the  most  important 
distant  location  requiring  personal  interviews.'  HQ  AFESC  is 
the  initiator  and  office  of  primary  responsibility  for  the 
Civil  Engineering  WANG  computer  system.  Personnel  at  HQ 
AFESC  were  instrumental  in  the  determination  of  a  MIS 
structure  best  suited  for  integration  on  the  WANG  and  use  by 
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Civil  Engineering  construction  managers.  HQ  AFESC  personnel 
also  explained  how  application  for  the  WANG  system  are 
typically  developed.  Personnel  operating  the  Naval  Aviation 
Lessons -Learned  database  system  also  provided  significant 
insight  into  the  development  of  DOD  on-line  lessons- learned 
systems . 

Telephone  interviews  were  conducted  with  the  AFRCEs  to 
formulate  the  underlying  user’s  requirements  data  needed. 
Although  personal  interviews  may  have  yielded  additional 
data,  the  cost  of  travel  outweighed  the  limited  additional 
benefits  of  more  data.  The  use  of  interviews,  both  personal 
and  telephone,  does  require  more  time  than  other  survey 
techniques.  Many  more  users  could  have  been  reached  in  an 
equivalent  amount  of  time  if  a  written  questionnaire 
approach  had  been  chosen.  The  use  of  written  questionnaires 
was  rejected,  because  the  goal  of  the  research  was  to  gather 
new  ideas.  Interviewing  offers  the  best  means  of 
accomplishing  gathering  new  ideas  (Davis,  1988).  Also,  the 
depth  and  breadth  of  information  obtained  through  interviews 
far  exceeds  that  gained  through  written  questionnaires 
(Emory,  1985:  160). 

Suiqmary 

A  review  of  books,  periodicals  and  journals  answered 
the  first  research  question,  "How  are  generic  management 
information  systems  developed?"  With  this  background 
understanding,  more  focused  research  answered  the  second  and 
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third  research  questions,  "What  lessons -learned  management 
information  systems  are  in  use;  and,  how  were  they 
developed?"  and  "What  common  factors  made  the  existing 
systems  successful  or  less  than  successful?".  The  research 
of  these  questions  involved  1 )  the  review  of  additional 
literature  including  operating  instructions,  user’s  guides, 
preparation  guides  and  assorted  system  specific  documents, 

2)  personal  and  telephone  interviews  with  the  developers  and 
operators  of  several  systems,  and  3)  direct  use  and 
observation  of  the  systems.  Finally,  answering  the 
remaining  research  questions,  “Bow  should  a  WANG -based 
lessons-learned  system  be  developed?"  and  "From  the  user’s 
perspective,  how  should  the  WANG  lessons-learned  MIS  be 
structured?"  required  personal  and  telephone  interviews  with 
Civil  Engineering  WANG  systems  personnel  and  AFRCE 
construction  managers. 

The  final  products  of  this  research  are  1)  a  generic 
methodology  that  should  be  followed  when  attempting  to 
initiate  a  lessons-learned  MIS  on  the  WANG  system  and  2)  a 
specific  description  of  system  characteristics,  interfaces 
and  functional  relationships  desired  by  AF  construction 
managers .  These  products  are  provided  in  Chapters  4  and  5 . 
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III.  Lit.erat.ure  Review 


Overview 

Every  study  is  a  search  for  information  (Emory, 
1985:135).  In  this  study,  the  search  is  for  information 
about  the  general  area  of  information  sharing  through  the 
use  of  management  information  systems  and  the  specific  area 
of  on-line  lessons-learned  databanks.  More  confidence  can 
be  placed  in  quality  of  the  findings  of  a  study  if  all 
sources  relevant  to  the  subject  have  been  explored  (Emory, 
1985:135). 

To  increase  the  confidence  in  the  findings  of  this 
study,  this  chapter  provides  a  review  of  previous  studies 
and  available  techniques  relevant  to  solving  the  research 
objectives.  The  structure  of  this  chapter  is  from  the  broad 
area  of  generic  management  information  systems  and  on-line 
information  sharing  to  the  focused  areas  of  MIS  development 
methodologies,  and  a  review  of  what  the  DOD  is  doing  in  the 
area  of  lessons-learned.  With  the  knowledge  gained  in  the 
broad  subject  review,  and  the  specific  strengths  and 
weaknesses  learned  in  the  focused  area  review,  a  clearer 
identification  of  the  problem  and  variables  involved  was 
accomplished . 

Previous  Studies 

Management  Information  Systems.  From  a  very  broad 
perspective,  "a  system  is  simply  a  set  of  components  that 
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interact  to  accomplish  some  purpose"  (Senn,  1984:11).  A 
management  information  system  (MIS)  then,  is  a  set  of 
components  that  provide  information  to  management.  These 
components  can  range  from  a  simple  organized  set  of 
documents  to  a  sophisticated  computer  system  accessible  to 
not  only  the  organization  owning  the  management  information 
system,  but  also  entities  external  to  the  owning 
organization  (Senn,  1984:  Chi). 

The  use  of  the  term  "information"  in  defining  or 
describing  MIS  is  what  distinguishes  information  systems 
from  simple  computers  or  software  programs  (Lucas,  1986: 

10).  Generally  speaking,  computers  and  their  associated 
software  programs  process  raw  data  and  then  output  that  same 
data  in  another  form  (Norton,  1986:  10-15).  The  difference 
between  information  and  data  is  in  the  user’s  perspective. 
Information  is  raw  data  that  has  been  processed  in  some  way 
to  allow  action  by  the  user  (Lucas,  1986:  11). 

An  example  of  this  distinction  can  be  seen  in 
considering  a  typical  telephone  book.  If  the  names  and 
numbers  in  a  telephone  book  were  listed  in  a  random  fashion, 
they  would  be  considered  raw  data.  However,  once 
alphabetized  and  sorted  by  city  and  state ,  the  raw  data 
becomes  information  because  it  allows  action  on  the  part  of 
the  user  (the  user  can  now  call  a  specific  person).  Lucas 
provides  a  schematic  representation  of  an  information  system 
similar  to  that  shown  in  Figure  3. 
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Figure  3.  Schematic  Representation  of  an 
Information  System  (Lucas,  1986:  11) 


The  definition  of  an  MIS  adopted  here  encompasses  a 
broad  area  of  data  processing.  For  this  reason,  the 
definition  of  management  information  systems  will  be  further 
narrowed  by  taking  a  look  at  the  concept  of  information. 
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Information  -  Differing  Systems  and  Needs.  From 
the  schematic  representation  of  an  MIS  shown  in  Figure  3, 
the  ultimate  use  of  an  MIS  is  to  provide  information  to  the 
user  which  will  allow  decision-making  processes  to  occur 
(Lucas,  1986:11).  Decision  making  processes  at  various 
organizational  levels  have  distinctly  different  information 
needs.  In  order  to  meet  these  differing  needs,  each  level 
has  evolved  its  own  type  of  information  delivery  tools 
(Powers  and  others,  1984:  9-10). 

As  stated  by  Lucas,  "For  most  organizations  -  in  the 
future,  if  not  already  -  the  determining  factor  in 
competition  will  be  the  processing  and  analysis  of 
information"  (Lucas,  1986:  5).  Senn  describes  information 
systems  as  "pervasive"  entities  depended  upon  to  some  degree 
by  all  organizations.  Further,  information  systems  link  an 
organization  together  "in  such  a  way  they  can  effectively 
work  toward  the  same  purpose"  (Senn,  1984:  11). 

The  lowest  level  of  information  needs  allows  control 
over  an  organization’s  routine  activities  and  transactions. 
This  information  need  is  fulfilled  through  Electronic  Data 
Processing  (EDP).  Essentially  nothing  more  than  an 
automation  of  existing  paperwork  procedures,  the  basic 
features  of  EDP  stress  a  focus  on  data  storage  and  retrieval 
at  the  operational  level  and  an  efficient  transaction 
processing  system.  It  is  generally  agreed  that  EDP  supports 
the  lowest  functional  levels  of  an  organization.  However, 
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EDP  does  provide  the  essential  information  data  base  for  all 
the  higher  level  information  systems  (Curtis,  1985:  20). 

The  second  tier  of  information  needs  in  an  organization 
is  the  ability  to  quickly  review  daily  transactions  and 
highlight  problem  areas  or  trends  needing  management 
attention.  It  is  this  level  of  information  needs  that 
management  information  systems  were  developed  to  support. 
According  to  Sprague  and  Carlson,  MIS  systems  have  a  middle 
manager  information  focus,  provide  very  structured 
information  flows,  and  allow  inquiry  and  report  generation 
with  EDP  data  bases  (Sprague  and  Carlson,  1982:  7). 

Higher  levels  of  information  needs  in  organizations 
also  exist,  for  example  information  needed  to  set  and 
achieve  long  range  strategic  goals  (Curtis,  1985:  20). 

These  information  needs  are  met  by  Decision  Support  (DSS) 
and  Expert  Systems.  Decision  Support  Systems  focus  on  the 
informational  needs  of  the  highest  levels  of  an 
organization.  DSS  utilize  the  results  of  EDP  and  MIS  and 
may  include  additional  data  brought  in  from  external 
sources.  Decision  support  systems  are  structured  as 
interactive,  computer-based  programs  to  help  decision  makers 
solve  infrequent,  unstructured  problems  (Curtis,  1985:21). 

The  next  step  of  providing  information  to  users  above 
DSS  would  be  to  actually  provide  not  only  the  information 
necessary  to  make  the  decision,  but  the  decision  itself. 

This  is  precisely  what  expert  systems  were  designed  to 
provide.  Expert  systems  have  traditionally  been  defined  as 


interactive  computer  programs  incorporating  judgement, 
experience,  rules  of  thumb,  intuition,  and  other  expertise 
to  provide  knowledgeable  advice  about  a  variety  of  tasks 
(Maher,  1987:3). 

According  to  Barris-Stewart ,  "expert  systems  are 
considered  the  most  practical  application  of  artificial 
intelligence  to  date."  Artificial  intelligence  is  the 
substitution  of  computers  to  yield  the  knowledge  and 
expertise  normally  obtained  from  human  beings  (Barris- 
Stewart,  1988:32).  According  to  Maher,  expert  systems  are  a 
result  of  many  years  of  attempting  to  simulate  or  reproduce 
intelligent  problem  solving  behavior  in  a  computer  program 
(Maher,  1987:3).  The  classic  definition  of  an  expert  system 
is  a  computer  program  that,  given  a  certain  set  of 
information,  will  yield  the  same  solution,  recommendation, 
or  answer  that  a  human  expert  would  provide,  given  the  same 
set  of  information  (Riker  and  others,  1987:11). 

Summarizing,  EDP  systems  provide  detailed  data  while 
management  information  systems  provide  selective  information 
through  further  processing  of  the  EDP  data  (Curtis,  1985: 
19-23).  With  a  firm  understanding  of  a)  exactly  what  an  MIS 
is,  and  b)  what  level  of  information  an  MIS  is  designed  to 
provide,  consider  how  an  MIS  could  be  applied  to  the  con¬ 
struction  management  problem. 

MIS  Development.  There  are  many  methods  used  in  the 
development  of  MIS  systems.  The  two  most  common  approaches, 
and  the  ones  that  are  considered  here,  are  the  traditional 
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and  user-driven.  Also  addressed  here  is  the  concept  of 
prototyping  -  a  commonly  used  MIS  development  tool. 

Traditional  Approach.  The  traditional  systems 
development  life  cycle  consists  of  se\en  activities:  1) 
preliminary  investigation,  2)  requirements  determination,  3) 
prototype  development,  4)  system  design,  5)  software 
development,  6)  systems  testing  and  7)  implementation  (Senn, 
1984:  18).  Although  several  of  these  activities  may  happen 
concurrently,  the  traditional  systems  development  cycle  is 
often  a  very  lengthy  process  (Went land,  1989). 

The  traditional  development  approach  saw  the  MIS 
designer  asking  the  managers  what  information  they  would 
like  to  have.  Unfortunately  for  the  MIS  designer,  most 
managers  did  not  really  know  what  information  was  needed. 
This  discovery  frequently  led  to  the  failure  of  the 
developed  MIS.  In  some  cases  executives  with  strong 
personalities  did  make  firm  statements  about  the  information 
needed  by  their  departments .  Systems  developed  for  these 
managers  were  usually  highly  personalized  and  almost  never 
survived  the  departure  of  the  user  they  were  created  for 
(Martin,  1984:  42). 

The  widespread  use  of  computers  created  a  growing 
demand  for  applications  and  a  shortage  of  programmers  to 
develop  new  systems.  As  a  result,  user-created  applications 
are  now  being  developed.  Another  more  powerful  reason 
driving  user- developed  systems  is  in  many  situations  the 
conventional  development  process  does  not  work.  All  too 


32 


frequently  systems  have  been  installed  after  years  of 
development  effort  only  to  result  in  the  end  users  saying  it 
is  not  what  they  want.  Even  worse,  in  many  instances  the 
users  try  a  system  for  a  while  and  then  give  up  because  they 
wanted  something  different  (Martin,  1984:  41).  Martin 
further  noted, 

A  common  reaction  to  this  unfortunate  situation  is 
to  say  that  the  requirements  were  not  specified 
sufficiently  thoroughly.  So  more  elaborate 
procedures  have  been  devised  for  requirements 
specification,  sometimes  resulting  in  voluminous 
documentation.  But  still  the  system  has  been 
unsatisfactory.  (Martin,  1984:  41) 

Many  organizations  have  made  attempts  at  getting  the 
application  creation  process  working  to  the  satisfaction  of 
the  users.  Often,  these  attempts  made  the  situation  worse. 
Steps  were  often  taken  to  enforce  more  formal  procedures  to 
convert  the  application  creation  process  "from  a  sloppy  ad 
hoc  operation  to  one  that  follows  rules  like  an  engineering 
discipline”  (Martin,  1984:42). 

The  DOD  recognized  it  had  problems  due  to  the 
traditional  software  development  process  and  mandated 
certain  actions  in  response  to  them,  in  DOD  Directive 
5000.29,  Management  of  Computer  Resources  in  Ma.ior  Defense 
Systeaa.  DOD  Directive  5000.29  attempted  to  reverse  the 
trend  of  programs  being  created  that  did  not  meet  the  user’s 
requirements.  A  Computer  Resource  Life  Cycle  Management 
Plan  specified  more  formal  requirements  documentation  prior 
to  the  design,  coding,  and  testing.  By  formalizing  the 
stages  of  systems  development  and  requiring  certain 
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milestones  be  attained  and  documented  the  DOD  hoped  to 
"ensure  the  proper  sequence  of  analysis,  design, 
implementation,  integration,  test  deployment  and 
maintenance"  (Martin,  1984:  42).  Within  the  DOD,  and 
elsewhere,  a  strong  push  has  been  made  to  "get  the  user  into 
the  driver’s  seat  of  systems  development"  to  improve  the 
development  process  (Wentland,  1989). 

Martin  uses  the  terms  “user-driven  computing  and 
prespecified  computing"  to  explain  the  distinctions  between 
the  traditional  approach  and  the  user-driven  approach  to 
systems  development.  These  distinctions  are  highlighted  in 
Figure  4  below. 

User-Driven  Approach.  The  user-driven  approach 
allows  fast  application  creation  because  whenever  possible, 
the  end  users  create  and  modify  their  own  applications.  The 
use  of  "fourth  generation"  computer  languages  has  made  it 
possible  to,  in  essence,  use  computer  software  to  help  write 
new  software.  These  application  generators  have  reduced 
systems  development  time  to  days  or  weeks.  In  other  cases  a 
system  analyst  using  applications  generators  creates  the  new 
application  working  at  a  terminal  in  concert  with  the  end 
user.  The  user-driven  process  is  incremental  and 
interactive,  as  the  process  uses  prototypes  to  replace 
lengthy  written  requirements  documents.  If  the  prototype 
meets  the  user’s  needs  it  is  often  converted  directly  into 
the  application  (Martin,  1984:  41-43). 
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Figure  4.  Systems  Development  Methodologies 
(Martin,  i984:  43-44) 


The  maintenance  of  user-driven  systems  is  almost 
continuous  as  a  result  of  the  incremental  change  process 
used.  Documentation  of  user-driven  systems  is  self 
generating,  as  the  application  is  created  by  the 
applications  generator  (Martin,  1984:  43-46).  At  its  best, 
the  user-driven  approach  is  "very  impressive  compared  with 
the  traditional  DP  development  cycle"  (Martin,  1984:  41). 
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Decentralization  of  information  systems  away  from  large 

centrally  located  mainframes  to  the  newly  developed  "small 

multi-user  minicomputer  systems  and  then  to  the  single  user 

microcomputers"  aided  the  trend  to  user-driven  application 

development  (Lee,  1988:  17).  In  many  instances. 

End  users  gained  control  of  much  of  applications 
development,  making  the  applications  quicker  to 
develop,  reducing  the  applications  development 
burden  on  the  DP/MIS  department,  and  greatly 
increasing  end  user  job  satisfaction. 

(Lee,  1988:  19) 

On  the  negative  side,  there  are  problems  with  the  user- 
driven  systems  approach.  By  nature,  user-developed  systems 
do  not  generally  follow  the  rigorous  development  discipline 
seen  in  the  traditionally  developed  systems.  Consequently, 
data  validation,  audit  trails,  backup  procedures  and 
documentation  are  often  lacking.  DP/MIS  departments  and 
their  information  systems  personnel  should,  as  much  as 
possible,  become  information  systems  consultants  to  the 
users  to  preclude  these  problems  (Lee,  1988:  18). 

Prototyping .  Prototyping  is  a  tool  that  can 

be  used  within  either  the  traditional  or  user-driven 

development  methodologies.  According  to  Martin, 

The  concept  of  prototyping  is  particularly 
important.  With  most  complex  engineering  a 
prototype  is  created  before  the  final  product  is 
built.  This  is  done  to  test  the  principles, 
ensure  that  the  system  works  and  obtain  design 
feedback  which  enable  the  design  to  be  adjusted 
before  the  big  money  is  spent.  . . .  Complex  data- 
processing  systems  need  prototyping  more  than  most 
engineering  systems  because  there  is  much  to  learn 
from  a  pilot  operation  and  many  changes  are  likely 
to  be  made.  (Martin,  1984:  46) 
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Until  the  1980s  the  cost  of  programming  a  prototype  was 
almost  equivalent  to  the  cost  of  programming  the  live 
working  system.  The  1980’s  brought  fourth  generation 
computer  languages  that  enabled  prototypes  to  be  created 
cheaply  and  quickly  (Kraushaar  and  Shirland,  1985:  189). 
Additionally,  packaged  software  programs  allowed  the  user  to 
experiment  and  try  different  methods  of  data-base  queries, 
report  generation,  and  manipulation  of  screen  information 
(Martin,  1984:  44). 

An  Application  of  MIS 

The  problem  addressed  in  this  research  could  be  reduced 
to  one  of  information  availability.  Specifically,  many  US 
Air  Force  construction  managers  are  inexperienced  and  unable 
to  easily  draw  on  the  experiences  of  others  in  their  field. 
Using  the  schematic  representation  of  the  various  components 
of  an  MIS  (shown  in  Figure  3)  as  an  outline,  the 
applicability  of  a  management  information  system  to  this 
problem  was  explored. 

Data.  The  data  that  is  needed  for  this  MIS  consists  of 
both  technical  and  managerial  " lessons-learaed"  in 
construction  management.  These  “lessons-learaed"  could  be 
something  as  simple  as  "avoid  the  use  of  flat  paint  on 
handrails  and  stair  casings  because  it  collects  dirt  easily 
and  is  difficult  to  maintain",  or  something  as  complex  as 
"the  use  of  an  asphaltic  waterproofing  membrane  between  the 


mudslab  and  structural  concrete  sections  on  an  inclined 


surface  requires  bracing  the  structural  sections  against  in 
situ  material  to  prevent  slippage.  This  is  required  because 
the  asphaltic  membrane,  under  pressure,  loses  viscosity  and 
acts  as  a  lubricant" .  This  latter  example  actually  occurred 
at  three  different,  large  scale  construction  sites  in  Saudi 
Arabia  because  the  "lessons-leamed"  at  one  site  were  not 
readily  available  to  the  other  sites  (Meister,  1989). 
Although  the  examples  given  here  are  primarily  technical  in 
nature,  management  lessons-leamed  are  also  applicable  to 
the  problem  and  could  be  input  into  the  MIS.  Management 
lessons  address  program  decisions,  budget  and  financial 
matters,  contracting  techniques,  maintenance  concepts  and 
data  management  (Schmidt,  Undated:  ii) 

Data  Collection.  Air  Force  Regulation  89-1,  requires 
construction  managers  to  “identify  positive  or  negative 
lessons-leamed"  during  facility  construction.  AFR  89-1 
also  directs  that  Post-Occupancy  evaluations  be  accomplished 
9  to  11  months  after  construction  completion  to  identify 
additional  lessons-leamed  (AFR  89-1,  1978:  4.5).  The 
primary  source  of  this  data  would  be  Air  Force  Regional 
Civil  Engineers  (AFRCE) . 

Processing.  Processing  of  data  into  a  usable  format 
would  require  the  use  of  some  form  of  organization  to  allow 
rapid  retrieval  by  subject,  engineering  discipline,  key 
words,  and  many  other  potential  inquiry  formats.  A  computer 
based  management  information  system  to  store  and  retrieve 
the  data  could  handle  the  expected  explosion  of  information 
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and  quickly  process  the  large  amounts  of  data  involved 
(Lucas,  1986:  11). 

Output.  Information  and  the  Oser.  The  output , 
information  and  user  subprocess  areas  of  an  MIS  are  combined 
here  to  emphasize  the  interaction  between  the  three 
elements . 

Perhaps  the  most  significant  aspects  of  MIS 
failure  involve  the  human  resource.  It  is, 
difficult  in  fact,  to  list  any  cause  of  failure 
that  does  not  have  its  origins  in  some  kind  of 
human  interface  with  the  machine. 

(Garrison,  1980:  15) 

The  importance  of  Garrison’ 3  statement  is  underscored 
when  it  is  recalled  that  information  is  determined  from  the 
user’s  perspective  (Lucas,  1986:  11).  If  the  output  does 
not  provide  information  to  the  user,  the  MIS  will  fail 
(Garrison,  1980:  12).  In  the  construction  management  MIS 
example,  the  form  of  the  output  must  be  determined  by  the 
construction  managers  who  are  relying  on  the  system  for 
information . 

Decision.  The  final  subprocess  area  of  an  MIS,  the 
decision,  hinges  on  the  value  of  the  information  provided. 
Information  has  key  characteristics  including  time  frame, 
expectation  value,  scope,  source,  frequency,  organization 
and  accuracy  (Lucas,  1986,  31).  For  construction  managers 
using  the  proposed  MIS,  all  of  these  attributes  of 
information  will  play  a  factor  in  whether  or  not  the 
information  provided  affects  the  decisions  required  in  their 
job.  If  the  information  is  accurate,  timely,  organized,  and 
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adequately  scoped,  it  will  affect  their  decision  and 
hopefully  provide  the  information  they  once  lacked  due  to 
limited  construction  experience. 

SMamarz 

Management  information  systems  provide  information  to 
management.  Seven  subareas  of  an  MIS  are  data,  collection, 
processing,  output,  information,  user  and  decision.  Working 
together  these  MIS  subareas  highlight  problem  areas  or 
trends  needing  management  attention.  Other  levels  of 
organizational  informational  needs  are  met  through 
electronic  data  processing,  decision  support  systems  and 
expert  systems . 

Two  common  approaches  to  the  development  of  MIS  systems 
are  the  traditional  and  user-driven  methods.  The 
traditional  systems  development  cycle  consists  of  seven 
activities:  1)  preliminary  investigation,  2)  requirements 
determination,  3)  prototype  development,  4)  system  design, 

5)  software  development,  6)  systems  testing  and  7) 
implementation.  The  advent  of  fourth  generation  computer 
languages,  and  problems  inherent  in  the  traditional 
approach,  have  led  systems  development  away  from  the 
traditional  approach  and  towards  the  user-driven  approach. 
The  user-driven  approach  is  less  structured,  quicker  and 
more  apt  to  result  in  a  system  that  meets  the  user's  needs 
than  the  traditional  development  approach.  With  the  user- 
driven  approach,  the  end  users  often  create  and  modify  their 
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own  applications  either  with  or  without  the  assistance  of  a 
systems  analyst.  Prototyping  is  a  tool  used  in  both  the 
user-driven  and  traditional  MIS  development  approaches. 
Regardless  which  development  methodology  is  used,  management 
information  systems  are  well  suited  to  the  basic  information 
sharing  problem  of  AF  construction  managers  needing  access 
to  the  lessons- learned  by  others. 

Having  considered  the  basics  of  management  information 
systems,  and  typical  MIS  development  methodologies  -  the 
next  chapter  reviews  existing  information  systems,  both 
general  and  in  the  specific  area  of  lessons-learned. 


IV.  Findings  Related  to  Existing  Systems 


Overview 

In  addition  to  the  information  gained  from  the 
literature  review,  significant  insight  into  information 
systems  was  obtained  by  using  several  existing  systems 
firsthand  and/or  talking  to  the  systems  operators.  This 
chapter  outlines  several  of  the  systems  investigated.  The 
selection  of  systems  to  review  began  with  general 
information  systems  and  progressed  to  lessons-learned 
systems.  The  degree  of  detail  provided  in  this  chapter  for 
general  systems  is,  by  intent,  less  than  that  provided  for 
the  lessons-learned  systems. 

General  Information  Systems 

Electronic  Bulletin  Boards.  The  proliferation  of 
personal  computers  (PC)  brought  with  it  the  proliferation  of 
electronic  bulletin  board  systems  (BBS).  PC  based  BBSs  are 
relatively  low  cost  systems  for  information  sharing  because 
they  can  be  set  up  on  any  personal  computer  with  a  hard  disk 
and  modem.  A  typical  bulletin  board  consists  of  a  PC,  one 
or  more  modems,  data  and  a  software  application  to  control 
the  use  of  that  data  by  remote  users .  The  owner  of  the  PC 
or  BBS  is  usually  the  systems  operator  (SYSOP) .  The  SYSOP 
keeps  the  bulletin  board  operational,  provides  help  to  new 
users,  and  generally  controls  what  the  bulletin  board  is 
used  for.  Remote  users,  through  their  PC  and  modem,  can 
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connect  to  the  bulletin  board  via  telephone  and  have  access 
to  whatever  information  is  on  the  BBS.  Most  BBS’  function 
as  a  place  to  trade  computer  programs,  play  games,  or  in 
general,  share  information  with  other  remote  users 
(Grimsley,  1989).  Several  bulletin  boards  were  explored, 
the  one  that  is  detailed  here,  Exec-PC  BBS  Network,  is 
fairly  representative  of  this  type  of  information  system. 

Exec-PC.  The  Exec-PC  BBS  Network  is  for  IBM-PC 
and  Compatibles,  UNIX/XENIX  Apple  Macintosh,  Commodore 
Amiga,  and  Atari  ST  computers.  Extensive  information, 
programs  and  files  for  these  computers  are  available  on  the 
Exec-PC  BBS.  Since  its  inception  in  1982,  Exec-PC  has  had 
almost  two  million  callers.  Exec-PC  is  used  by  more  than 
2,000  callers  per  day  searching  for  free  software  or  message 
system  activity  related  to  their  interests  (Exec-PC,  1989). 

One  typical  problem  of  BBSs  is  access.  Users  often  can 
not  log  onto  a  BBS  because  the  phone  lines  are  busy.  Exec- 
PC  has  nearly  eliminated  busy  signals  by  using  over  90  phone 
lines.  Although  actually  a  very  large  system,  Exec-PC  is 
very  user-friendly.  Easy,  self  explanatory,  direct  menus 
with  full  graphics  capability  using  color  and  graphics  to 
guide  your  eye  makes  Exec-PC  simple  to  understand  and  use. 
Exec-PC’ s  interface  is  logical  in  its  layout,  as  shown  by 
the  Exec-PC  main  menu  in  Figure  5  (Exec-PC,  1989). 

Exec-PC  allows  2  levels  of  access.  Free  access  gives 
limited  use  of  the  system,  paid  access  gives  full  use  of 
the  system.  Full  paid  access  allows  the  user  420  minutes  of 
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Exec -PC  TOP  M  E  N  0 


<?>help .  HELP  with  this  aenu 

<S>ubscribe  . . .  Subscribe/Renew  for  full  access 
<B>ulletins  . . .  Info  about  this  BBS 

<H>elp  .  HELP  on  most  often  asked  questions 

<R>ead  sail  . . .  Read  all  pending  messages  for  me 

<F>ile  .  File  Collections 

<M>essage  .  Message  system 

<E>nvironment  .  Change  password,  defaults,  etc. 

<A>nsi/color  . .  Turn  on/off  color  and  graphics 
<L>ist  users  . .  Info  on  other  users 

<W>ho  .  Who  is  on  the  system  right  now? 

<X>pert . Expert  mode  (short  or  long  menus) 

<G>oodbye  .  Log  off  system  (hang  up) 


(29  minutes  left)  TOP  ( SBHFMREALWXG ,  ?=HELP)  -> 


Figure  5.  Typical  Bulletin  Board  Menu 
(Exec-PC,  1989) 


BBS  log-on  time  per  week  and/or  a  download  limit  of  four 
megabytes  (4,000,000)  of  files  per  week.  Additionally,  full 
access  users  can  send  messages  to  anyone  in  all 
conference/topics  areas,  send  and  receive  private  email 
to/from  anyone,  read  all  messages  in  all  public  conferences, 
download  files  from  all  file  collections,  and  upload  files 
to  all  collections  in  which  upload  is  an  option.  Users  who 
don’t  pay  to  join  Exec- PC  are  limited  to  30  minutes  per  day 
access  time,  during  which  they  may  send/read  messages 
to/from  the  SYSOP,  read  all  messages  in  the  MS-DOS 
conference  and  download  up  to  380,000  bytes/day  from  the 
free  file  collections.  Exac-PC  rates  are  $20  for  3  months 
full  access  or  $60  for  one  year  full  access  (Exec-PC,  1989). 
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As  noted  by  the  opening  bulletins,  Exec-PC  supports, 

Ninety  incoming  phone  lines,  24  hours  per  day. 

All  lines  have  2400  baud  modems  with  MNP  error 
correction.  USE  HST  dual  standard  14,400  baud  & 

7.32  modems  on  some  lines.  Over  70,000  total 
files  collected  in  the  Mahoney  and  PC-SIG  systems, 
plus  a  4,200  megabyte  (4.2  gigabyte)  high  speed 
mass  storage  system.  Numerous  message  conferences 
and  topics  supported  by  an  advanced  software 
system.  (Exec-PC,  1989) 

In  addition  to  the  features  noted,  Exec-PC  uses  the 
Hyperscan(tm)  file  search  feature,  capable  of  searching 
20,000  file  entries  in  less  than  2  seconds.  Exec-PC  users 
also  have  direct  connect  to  the  Telenet(tm)  nationwide  data 
network  (Exec-PC,  1989).  These  and  other  features  combined 
make  Exec-PC  a  very  useful  information  sharing  tool. 

Research  Systems.  There  are  many  on-line  information 
systems  designed  to  aide  research  in  scientific  and 
technical  areas.  Some  provide  access  to  a  broad  spectrum  of 
research  areas,  others  are  designed  to  support  one  specific 
area  of  research.  One  of  the  largest  research  oriented 
information  systems  is  the  Defense  Research  Development  Test 
and  Evaluation  On-Line  System  (DR0LS)  operated  by  the 
Defense  Technical  Information  Center  (DTIC). 

DROLS  System.  The  DROLS  system  is  the  on  line 
portion  of  the  Defense  Technical  Information  Center.  The 
mission  of  DTIC  is  to  further  Department  of  Defense  research 
and  development  (R&D)  efforts  by  increasing  the  access  and 
transfer  of  scientific  and  technical  information  applicable 
to  defense  R&D.  DROLS  is  one  tool  DTIC  uses  to  meet  this 
mission.  DROLS  has  access  to  over  1.2  million  technical 
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reports,  notes  and  memoranda  contained  in  four  separate 
databases  (Taylor,  1989). 

DROLS  data  is  obtained  through  the  principle  users  of 
DROLS  -  DOD  agencies,  DOD  contractors,  government  agencies, 
educational  institutions  and  foreign  agencies  and 
institutions.  Osers  of  DROLS  may  access  the  on-line 
databases  using  either  a  personal  computer  and  modem  or 
UNIVAC  terminals  hard-wired  to  DTIC’s  central  computer 
system.  Classified  information  is  only  accessible  using 
dedicated  phone  lines  and  hard-wired  terminals  (DTIC,  1985). 

DROLS  has  been  on-line  since  1968  and  is  hosted  on  a 
Sperry  /Uni-vac  (UNISYS)  computer  system.  There  are  1080 
users  of  DROLS;  however,  a  single  DROLS  user  is  most  often 
an  agency  or  institution  (such  as  the  AFIT  library)  that 
performs  DROLS  searches  for  many  people  (Taylor,  1989). 

DROLS  users  can  search  the  databases  for  information  based 
on  author,  source,  report  date,  title  or  subject.  DROLS 
supports  text  searching  of  the  title  and  report  narrative  on 
a  limited  basis  (DTIC,  1985). 

Because  of  the  costs  associated  with  connection  to 
DROLS  ($30  per  connect  hour),  on-line  research  of  the  DROLS 
was  very  limited.  However,  other  research  oriented  systems 
were  researched  intensively.  Specifically,  the  Lunar  and 
Planetary  Institute’s  PATRON  System,  US  Department  of 
Agriculture’s  BIONET  System  and  the  National  Oceanic  and 
Atmospheric  Administration’s  Meteorological  Database  System, 
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were  researched.  Of  these  systems,  the  PATRON  and  BIONET 
are  also  presented  here. 

PATRON  System.  The  Lunar  and  Planetary  Institute 
(LPI)  in  Houston  Texas,  operates  an  on-line  information 
system,  PATRON,  to  make  the  space  research  material  in  their 
library  information  center  more  accessible  to  more  people. 
The  PATRON  System  offers  on-line  access  to  all  basic  library 
functions  such  as  bibliographical  searches,  card  catalog 
searches,  reference  material  ordering,  etc.  PATRON  provides 
access  to  the  library  when  a  staff  member  is  not  available 
(after  hours  St  weekends)  or  from  a  remote  location.  Through 
PATRON  new  users  can  also  learn  about  the  library 
information  center  and  the  various  services  that  are 
offered,  quickly  and  easily.  The  primary  functions  of 
PATRON  can  be  realized  by  looking  at  PATRON’S  main  menu, 
shown  in  Figure  6  (PATRON,  1989). 

According  to  Stephen  Tellier,  responsible  for  the 
development  and  initiation  of  the  PATRON  system,  PATRON  is 
hosted  on  a  VAX/1170  computer  at  the  Institute.  PATRON  is 
accessible  via  direct  connect  terminals  or  remote  terminals 
via  modem  at  300/1200/2400  BAUD  transmission  rates.  PATRON 
is  a  menu  driven  system,  but  the  menus  can  be  disabled  by 
users  who  prefer  to  operate  in  a  command  driven  mode.  The 
PATRON  on-line  help  facilities  make  using  the  system  very 
easy.  PATRON  has  seen  a  200  percent  increase  in  usage  in 
two  years  and  is  now  averaging  over  1000  calls  per  year. 

The  Lunar  and  Planetary  Institute  is  now  on  the  Space 
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MAIN  PATRON  PROGRAM  MEND 

Select  the  routine  you  wish  to  use 
by  entering  its  letter  below 

A.  EXPLAIN  LIC  SERVICES 

B.  SEARCH  THE  CARD  CATALOG 

C.  CHECK  ON  LATEST  ARRIVALS 

D.  CHECK  JOURNAL  HOLDINGS 

E.  PLACE  REQUESTS  FOR  MATERIAL 

F.  LEAVE  MESSAGES  FOR  LIC 

G.  QUIT  (EXIT  THE  ACCOUNT) 

H.  CALL  PATRON  HELP  ROUTINE 

ENTER  LETTER  CHOICE:  _ 

Figure  6.  PATRON  System  Main  Menu 
(PATRON,  1989) 

Physics  Analysis  Network  (SPAN)  and  an  additional  increase 
in  users  is  expected.  Users  of  the  patron  system  who  also 
have  access  to  SPAN  may  use  it  to  connect  with  other  LPI  on 
line  services  (Tellier,  1989). 

BIONET  System.  The  Biology  Network  (BIONET) 
System  is  an  on-line  molecular  biology  databank  for 
scientists  and  researchers.  BIONET  is  a  menu-driven  system 
open  to  all  non-profit  organizations  associated  with  USDA 
Agricultural  Research  Services  (ARS).  Through  BIONET, 
researchers  and  scientists  investigating  DNA/RNA  sequencing 
and  protein  and  nucleic  acid  sequencing  can  quickly  access 
the  work  of  others  in  their  field  to  determine  whether 
particular  sequences  are  new  or  merely  a  match  to  known 
sequences.  BIONET  provides  an  avenue  where  researchers 
working  with  cloning,  DNA/RNA  sequences,  disease  analysis 
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or  any  of  the  many  fields  in  plant  or  animal  molecular 
biology  areas  can  share  their  findings  and  increase 
technology  transfer  within  the  field  (Laster,  1989). 

BIOMET  users  can  also  access  the  OS  National  Institute 
of  Health's  (NIH)  Genetics  Databank  and  the  Molecular 
Biology  Laboratory’s  Databank  in  Heidelburg,  Germany. 

BIONET  is  established  in  Beltsville  Maryland  as  a  OSDA 
satellite  installation  through  a  cooperative  agreement 
between  the  National  Institute  of  Health  and  the 
Intelligenetics  Corporation  of  Palto  Alto,  California 
(Laster,  1989). 

Lessons -Learned  Management  Information  Systems 

In  the  specific  area  of  lessons-leamed,  several 
management  information  systems  have  been  developed  and  are 
in  use  today.  These  lessons- learned  MISs  range  from  manual 
systems  used  by  individual  organizations  to  computer-based, 
service-wide  systems.  Each  management  information  system 
has  individual  strengths  and  weaknesses  that  contribute  to 
its  respective  success  or  failure.  Two  of  the  largest  DOD 
lessons-learned  management  ’nformation  systems  -  the  Navy's 
Naval  Acquisition  Lessons -Learned  system  and  the  Air  Force’s 
On-Line  Access  Lessons-Learned  system  were  explored.  The 
D.S.  Army  Research  and  Development  Center  maintains  a  series 
of  lessons-learned  reports  available  in  hard-copy  format 
(Booker,  1989).  Although  in  the  process  of  developing  an 
on-line  lessons-learned  system,  the  Army  does  not  currently 
have  an  on-line  lessons-learned  system  (Grimsley,  1989). 
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In  1982  the 


Naval  Air  Systems  Command  (NAVAIRSYSCOM)  initiated  a 
lessons-learned  program  to  support  the  Joint  Services 
Advanced  Vertical  Lift  V-22  “Osprey"  aircraft  acquisition. 
The  Naval  Aviation  Lessons-Leamed  (NALL)  system  is  the 
result  of  that  effort  (Gardiner,  1989).  As  outlined  in  the 
Naval  Aviation  Lessons-Learned  Program  Management  Plan,  the 
goals  of  the  NALL  are  to  “reduce  procurement  and  life  cycle 
costs,  improve  reliability  and  maintainability,  and  improve 
readiness  by  learning  from  past  experiences"  (Naval  Aviation 
Lessons-Leamed  Program  Management  Plan,  1986:1). 

History.  The  lessons-learned  concept  was 
sponsored  through  a  tri-service  agreement  between  the  Joint 
Logistics  Commanders  (JLC)  of  the  Army  Material  Command, 
Chief  of  Naval  Operations,  Air  Force  Systems  Command  and  Air 
Force  Logistics  Command  (NALL  Preparation  Guide,  1989:  3). 
Although,  no  longer  within  the  JLC  arena,  the  NALL  continues 
to  operate  under  a  tri-service  memorandum  of  agreement 
(Naval  Aviation  Lessons-Leamed  Program  Management  Plan, 
1986:  2).  The  tri-service  memorandum  of  agreement, 
effective  15  March  1989,  serves  as  the  implementation 
directive  through  which  lessons-learned  exchange  procedures 
are  established  (Grimsley,  1989). 

The  initiation  of  the  NALL  began  with  the  collection  of 
lessons  from  a  variety  of  sources  including  the  Army  and  Air 
Force.  A  lessons-learned  team  also  interviewed  managers, 
supervisors,  operators,  and  maintenance  personnel  through  a 


50 


series  of  visits  to  fleet  operating  and  support  activities. 
These  interviews  concentrated  on  both  positive  and  negative 
information  about  the  supportability ,  maintainability,  and 
reliability  factors  of  current  weapon  systems.  The  material 
collected  was  then  evaluated  and  developed  into  potential 
lessons-learned  (NALL  Preparation  Guide,  1983:  1-2). 

The  resultant  lessons-learned  were  input  into  a 
computer  database  beginning  in  1983  and  completed  in  1984. 
The  initial  database  of  lessons-learned  was  less  helpful 
than  originally  envisioned,  primarily  due  to  the  difficulty 
experienced  by  users  attempting  retrieval  of  the  lessons- 
learned  (Grimsley,  1989).  This  early  version  of  the 
automated  NALL  was  extremely  “ user-unf r iendly " ,  provided 
minimal  information  through  a  series  of  one-page  reports, 
and  was  severely  limited  in  lessons-learned  manipulation  and 
extract  capabilities  (Grimsley,  1989). 

Configured  as  a  series  of  "quick  and  dirty  batch 
programs",  written  with  limited  input  from  the  user,  the 
early  NALL  system  did  not  meet  the  needs  of  the  user  (Dove, 
1989).  Consequently,  the  early  NALL  was  infrequently  used 
and  program  interest  “died  for  a  period  of  six  to  eight 
months"  (Grimsley,  1989). 

Upon  recognition  that  the  original  NALL  database  was  of 
limited  usefulness,  a  revised  NALL  system  was  considered  to 
overcome  the  weaknesses  cited.  Through  extensive 
interaction  and  communications  between  the  NALL  users  and 
system  programmers,  the  on-line  NALL  database  and  user 
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interface  was  restructured  to  its  current  configuration 
(Dove,  1989).  The  current  NALL  system  has  over  200  users 
compared  to  the  original  system’s  15  users.  Despite  this 
increase  in  usage,  computer  costs  associated  with  the  system 
have  decreased.  The  NALL  operators  attribute  this  drop  in 
cost  to  the  NALL’s  increased  user-friendliness  and  the  users 
resultant  ability  to  quickly  find  and  extract  only  the 
information  they  need  (Grimsley,  1989). 

One  potential  measure  of  the  current  NALL  system’ s 
success  is  the  revived  interest  in  the  program,  as  witnessed 
by  the  increased  usage.  Additionally,  several  contractors 
such  as  McDonnell  Douglas,  Boeing,  Hughes  Aircraft,  General 
Dynamics  and  others  have  either  initiated  their  own  lessons- 
learaed  system  or  requested  installation  of  the  NALL  on 
their  company  computers  (Grimsley,  1989). 

System  Conf iguration.  The  NALL  database  is 
installed  on  an  IBM  Amdahl  470VA  mainframe  computer  located 
in  the  Computer  Sciences  Directorate  at  the  Naval  Air  Test 
Center,  Patuxent  River,  MD.  The  on-line  system  operates 
under  the  Multiple  Virtual  System  (MVS),  Time  Sharing  Option 
(TSO),  and  System  2000  Database  Management  System  software 
(NALL  User's  Guide,  1983-  S). 

The  NALL  can  be  accessed  through  any  terminal  directly 
tied  to  the  Amdahl  mainframe.  Alternatively  the  NALL  is 
accessible  to  remote  users  possessing  a  user-id  and  password 
provided  by  the  Naval  Air  Test  Center  upon  request.  Remote 
users  must  have  an  IBM  personal  computer  or  equivalent,  with 
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a  modem  and  at  least  one  floppy  disk  drive  and  a  hard  drive. 
Prior  to  accessing  the  system,  remote  users  must  install  the 
HALL  Remote  User’s  Software  provided  by  the  Naval  Aviation 
Lessons-Leamed  Office.  The  HALL  is  available  for  access  24 
hours  per  day,  seven  days  per  week.  (NALL  User’s  Guide, 
1989:  4-5). 


NALL  Database  Configuration.  The  NALL  database 
currently  contains  1298  lessons  (Dove,  1989).  Each  lesson 
contains  the  ten  different  sections  listed  and  explained 
below : 

1.  NATO  CALL  NUMBER:  Internal  tracking  number  unique 
to  each  lesson  learned. 

2.  ACCESS  NUMBER:  Assigned  by  the  Lessons-Learned 
Research  Team  for  tracking  and  referencing  during 
validation. 

3.  IMPACT  AREAS:  A  listing  of  up  to  six  major  areas 
that  the  lesson  affects,  for  example:  safety,  engineering, 
facilities,  human  factors,  health,  training,  reliability, 
survivability,  etc.  There  are  44  total  impact  areas. 

4.  TOPIC:  Title/subject  of  the  lesson  learned,  brief 
but  representative  of  the  content  of  the  lesson. 

5.  LESSON  LEARNED:  The  actual  lesson  learned,  its 
cause  and  its  effect. 

6.  PROBLEM:  Descriptive  statement  of  what  went  wrong. 
If  the  lesson  is  positive,  this  area  says  "NONE”. 

7.  DISCUSSION:  Summary  of  the  lesson  learned  research 
findings  conducted  to  validate  the  lesson. 

8.  APPROPRIATE  ACTION:  Recommendation(s)  on  possible 
ways  to  avoid  the  problem. 

9.  WORK  UNIT  CODE  (WUC):  A  two-digit  code  used  to 
identify  the  specific  system  or  type  of  equipment  affected 
by  the  lesson.  Over  100  codes  are  available  in  the  Standard 
Work  Unit  Code  Manual,  however  only  up  to  five  are  used  on 
each  lesson.  WUC  examples  include:  helicopter  rotor 
system,  landing  gear,  support  equipment,  simulators,  power 
systems,  utilities,  etc. 


53 


10.  AIRCRAFT  TYPE:  A  two-letter  code  denoting  the 
type  of  aircraft  affected,  such  as  rotary  wing,  fixed  wing, 
fighter,  etc.  This  code  is  applied  only  if  the  lesson 
applies  only  to  that  type  of  aircraft.  If  the  lesson  is 
applicable  to  all  aircraft  or  non-aircraft  support  equipment 
then  the  codes  AA  or  SE,  respectively,  are  used. 

By  agreement  between  the  Joint  Logistics  Commanders, 
the  call  number,  topic,  lessons-learned,  problem,  discussion 
and  appropriate  action  sections  constitute  “the  minimum 
standard  format  for  documenting  individual  lessons-learned 
within  and  among  the  Services'*  (Joint  Agreement  on  the  JLC 
Lessons-Learned,  1989:  1-2). 

For  purposes  of  readability  and  conciseness,  typical 
NALL  lessons-learned  are  no  more  than  one  typed  page  long. 
One  of  the  principle  concepts  behind  lessons-learned  is 
simplicity.  Lessons  are,  and  should  be,  written  in  plain 
English  with  minimal  use  of  jargon  and  technical  terminology 
(Gardiner,  1989). 

The  primary  emphasis  of  the  NALL  is  lessons-learned  in 
the  acquisition  of  Naval  aviations  systems  or  aircraft, 
hence,  lessons-learned  in  construction  or  facility 
acquisition  are  not  directly  supported.  For  example, 
designators  such  as  "work  unit  code"  and  "aircraft  type"  are 
not  appropriate  for  facility  construction  lessons.  Although 
the  NALL  does  not  directly  support  construction  lessons,  the 
application  of  the  concept  of  construction  lessons-learned 
is  valid.  The  Navy  projects  its  power  through  ships,  not 
airfields  and  facilities  -  hence,  Navy  facilities  receive 
less  emphasis  than  Air  Force  facilities  (Grimsley,  1989). 
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A  transcription  of  a  sample  lesson  retrieved  from  the 


NALL  is  shown  in  Figure  7  below: 


CALL  NUMBERS:  IMPACT  AREA(S) : 

NATC  00271  Design 

ACCESS  88-793  Reliability 

Human  Factors 

TOPIC:  Helicopter  transmission  oil  fill  screens 

LESSON  LEARNED:  Lack  of  screens  on  helicopter 

transmission  oil  filler  necks  can  result 
in  foreign  object  damage  to  the 
transmission. 

PROBLEM:  Foreign  objects  cure  being  introduced  into 

helicopter  transmissions  during  oil 
servicing. 

DISCUSSION:  Some  helicopter  transmissions  do  not  have 

screens  in  the  oil  filler  neck.  Foreign 
objects  such  as  the  foil  screens  from 
oil  containers  have  been  introduced 
during  servicing.  Although  use  of 
servicing  units  incorporating  in-line 
filters  is  required,  unit  unavailability 
or  nonuse  by  servicing  personnel 
exposes  the  oil  system  to  possible 
introduction  of  foreign  objects. 

APPROP.  ACTION:  A)  Designers  of  helicopter  transmissions 

should  incorporate  screens  in  the  oil 
filler  necks  capable  of  preventing  entry 
of  foreign  objects  during  servicing. 

B)  Specifications  for  transmissions  and 
gearboxes  should  require  fod  screens  on 
oil  servicing  or  filler  openings. 

C)  Consideration  should  be  given  to 
retrofit  filler  necks  lacking  screens. 

D)  Increased  emphasis  must  be  placed  on 
training  to  ensure  proper  servicing. 

WUC:  15  AG  AP  AX 


Figure  7.  Sample  NALL  Lesson 
(NALL,  1989) 
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Data  Input.  Potential  lessons-learned  for  the 
NALL  are  obtained  from  a  variety  of  sources  including 
successful  or  unsuccessful  acquisition  program  experiences, 
personal  experiences,  test  reports,  inspection  deficiencies, 
safety  mishap  data,  maintenance  data,  engineering  data  and 
contractors.  Virtually  anyone  who  has  experience  in  naval 
aviation  weapons  systems  design,  acquisition,  operation  or 
maintenance  is  considered  a  source  for  lessons-learned 
(Grimsley,  1989). 

Potential  lessons-learned  are  submitted  to  the  Naval 
Air  Test  Center,  Rotary  Wing  Aircraft  Test  Directorate 
(RWATD)  for  research  and  validation.  The  primary  input  of 
potential  lessons  is  accomplished  by  an  in-house  staff  of 
researchers  who  have  set  up  contacts  with  field  units  and 
contractors.  Additionally  the  research  staff  monitors  a 
flow  of  written  communications  (safety  reports,  naval 
message  traffic,  readiness  summaries,  etc)  and  looks  for 
repetitive  trends  or  an  inordinate  amount  of  communications 
on  positive  and/or  negative  experiences  (Naval  Aviation 
Lessons-Learned  Preparation  Guide,  1989:  1-3).  Other 
lessons-learned  submissions  are  received  through  personal 
contact  during  lessons-learned  staff  visits  to  field  units 
and  contractors.  Lastly,  a  recent  source  of  input  is  the 
NALL  bulletin  board,  tj  be  discussed  later  (Grimsley,  1989). 

Once  a  potential  lesson  learned  has  been  identified  and 
forwarded  to  RWATD,  it  is  assigned  to  one  of  8-10  staff 
researchers.  The  researcher  contacts  the  individuals 
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involved  with  the  potential  lesson,  reviews  all  pertinent 
technical  data,  and  validates  that  the  potential  lesson  is 
more  than  an  opinion  or  result  of  failure  to  follow 
regulatory  guidelines.  Quality  research  is  critical  to  the 
validation  of  lessons -learned.  Accordingly,  a  potential 
lesson  will  not  be  researched  in-depth  or  logged  in  until  it 
is  supported  by  two  or  more  recognized  sources  (Naval 
Aviation  Lessons- Learned  Preparation  Guide,  1989:  5-6). 

Upon  completion  of  the  validation  the  researcher 
prepares  a  synopsis  of  the  research  and  analysis  in  a 
lessons- learned  format.  It  is  the  policy  of  the  Navy  to 
sanitize  all  lessons-learned  such  that  neither  the  lesson 
nor  the  backup  documentation  attached  contains  any  reference 
to  the  use  of  "specific  type /model  of  aircraft/equipment, 
contractors,  subcontractors,  or  vendors"  (Naval  Aviation 
Lessons -Learned  Preparation  Guide,  1989:  5-6).  This  policy 
is  established  both  to  preclude  legal  disputes  and  to 
encourage  those  involved  in  negative  lessons  not  to  be 
afraid  of  forwarding  a  lesson  learned  (Grimsley,  1989). 

After  review  and  approval  of  the  lesson  by  the  NALL  review 
board,  comprised  of  NALL  team  members  and  fleet  and  field 
engineers,  the  approved  lesson  is  entered  into  the  NALL 
database  (Naval  Aviation  Lessons-Learned  Preparation  Guide, 
1989:  7). 

Data  Retrieval.  The  NALL  system  initiators  made 
every  attempt  to  ensure  the  NALL  is  user  friendly.  The  NALL 
Is  a  menu-driven  system  allowing  the  user  to  select  which 
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activity  is  desired  from  a  preset  menu  of  options.  Each 
menu  is  self-explanatory  and  the  options  available  are 
clearly  shown.  The  layout  and  options  provided  by  each 
menu,  and  the  program-level  interface  between  submenus  were 
established  through  close  coordination  between  the  users  and 
system  programmers  (Dove,  1989). 

The  opening  menu  of  the  NALL  system,  shown  in  Figure  8 
below,  allows  the  user  to  select  either  a  REPORTS  or 
RETRIEVAL  search  method . 


NAVAL  AIR  TEST  CENTER 
RELIABILITY  AND  MAINTAINABILITY 
DATA  BASE  MASTER  MEND 


A  -  RETRIEVAL 
B  -  REPORTS 

C  -  UPDATE 

X  -  EXIT 


Figure  8.  NALL  Master  Menu 
(NALL,  1989) 


The  RETRIEVAL  search  method  allows  a  different  series 
of  category  searches  than  the  REPORTS  section  and  does  not 
generate  an  index  of  the  lessons  found  during  the  search. 

The  REPORTS  search  method  automatically  produces  an  index  of 
the  lessons-learned  found  during  the  search.  It  also  allows 
retrieval  by  NATC  number  and  can  build  a  series  of  abstract 
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reports  containing  only  the  lesson  learned  and  appropriate 
action,  or  the  topic  and  work  unit  codes  (Dove,  1989).  The 
REPORTS  and  RETRIEVAL  menus  are  shown  in  Figures  9  and  10, 
respectively . 


ROTARY  WING  TEST  DIRECTORATE  LESSONS  LEARNED 

REPORT  MEND 


A  -  GLOBAL  SEARCH 
B  -  WORK  UNIT  CODE  SEARCH 
C  -  NATC  NUMBER  SEARCH 
D  -  AIRCRAFT  TYPE  SEARCH 
E  -  IMPACT  AREA  SEARCH 

F  -  NATC  *  ABSTRACT  (LESSON  AND  APPROPRIATE  ACTION) 
G  -  WDC  ABSTRACT  (TOPIC  AND  WORK  UNIT  CODE) 

X  -  EXIT 

ENTER  OPTION: 


Figure  9.  NALL  Report  Menu 
(NALL,  1989) 


The  REPORTS  search  method  allows  either  category 
searches  or  global  searches.  The  RETRIEVAL  search  method 
allows  either  global  searches  or  searches  based  on  work  unit 
codes,  NATC  numbers,  aircraft  type  or  impact  area.  Global 
searches  scan  the  entire  lesson  for  the  chosen  keyword(s). 
Category  searches  scan  only  the  user  specified  section  of 
the  lesson  (appropriate  action,  problem,  etc. )  for  the 
chosen  keyword(s).  Both  global  and  category  searches  bring 


59 


ROTARY  WING  TEST  DIRECTORATE  LESSONS  LEARNED 
RETRIEVAL  MEND 


A  -  GLOBAL  SEARCH  F  -  LESSON  LEARNED  SEARCH 

B  -  WORK  UNIT  CODE  SEARCH  G  -  PROBLEM  SEARCH 

C  -  AIRCRAFT  SEARCH  H  -  DISCUSSION  SEARCH 

D  -  IMPACT  AREA  SEARCH  I  -  APPROP.  ACTION  SEARCH 

E  -  TOPIC  SEARCH 

X  -  EXIT 

ENTER  OPTION: 

Figure  10 .  NALL  Retrieval  Menu 
(NALL,  1989) 

the  NATC  number,  access  number,  and  topic  of  the  lesson  to 
the  screen  for  viewing  or  printing.  Alternatively,  an  index 
can  be  generated  for  display  consisting  of  the  NATC  number 
and  the  topic  and  page  number  of  the  lesson.  After  review 
of  the  index,  the  lessons  are  displayed. 

Keyword  searches  are  "and"  type  searches  which  require 
all  of  the  key  words  to  be  found  in  the  lesson  before  the 
lesson  will  be  retr  eved.  Before  beginning  keyword 
searches,  the  system  prompts  the  user,  ”D0  YOU  WANT  A 
COUNT?".  A  "Yes"  response  advises  the  user  how  many  lessons 
will  be  retrieved.  Depending  on  the  count  and  the  user’s 
needs,  the  search  can  be  made  narrower  or  broader  by  the 
keywords  specified.  The  more  keywords,  the  narrower  the 
search.  All  other  searches  are  "or"  type  searches  where 


60 


only  one  of  the  numbers  selected  must  match  for  a  lesson 
retrieval  to  occur  (NALL  User’s  Manual,  1989:  14-20). 

Data  Sorting  and  Retrieval .  Once  the  user  has 
selected  either  the  REPORTS  or  RETRIEVAL  search  methods  and 
pressed  the  menu  option  for  the  specific  search  desired,  the 
NALL  presents  a  SORT  OPTIONS  menu  to  allow  the  user  to  sort 
the  lessons  on  either  NATO  number,  work  unit  code  or 
aircraft  type .  After  the  search  has  been  initiated  from  the 
SORT  OPTIONS  menu,  the  NALL  provides  an  "X  SYSTEM"  prompt  at 
the  bottom  of  the  screen  to  indicate  that  the  retrieval 
and/or  report  is  being  generated  (NALL  User’s  Manual, 

1989:  21-59). 

If  the  retrieval  was  generated  through  the  REPORTS 
search  method,  completion  of  the  search  is  indicated  by  the 
appearance  of  an  index  on  the  screen.  The  user  may  then 
scroll  through  the  index  to  review  which  lessons  were 
generated  by  the  search.  Immediately  following  the  index 
are  the  lessons  themselves,  sorted  according  to  the  method 
selected  from  the  SORT  OPTIONS  menu.  If  the  retrieval  was 
generated  through  the  RETRIEVAL  search  method,  completion  of 
the  search  is  indicated  by  the  appearance  of  a  "TOP  OF  DATA" 
prompt  at  the  top  of  the  screen.  The  user  may  then  scroll 
through  the  lessons  retrieved  (NALL  User’s  Manual, 

1989:  60-61). 

Printing  Lessons-Leamed.  Hard-copies  of  the 
lessons  retrieved  and/or  reports  generated  are  easy  to 
obtain  from  the  NALL  system.  When  the  user  reviews  the 
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lessons  retrieved,  the  system  initiates  the  prompt,  "DO  YOU 
WANT  THIS  PRINTED?"  on  the  screen.  If  the  user  enters  “Y" 
(for  yes),  a  new  prompt  appears  offering  the  user  the  choice 
of  remote  or  local  printing.  Local  printing  refers  to 
printing  at  the  Naval  Aviation  Test  Center.  Selection  of 
"Remote"  printing  will  initiate  a  small  program  that 
transmits  the  lessons  from  the  Naval  Air  Test  Center  Amdahl 
mainframe  to  the  printer  at  the  remote  user's  location.  A 
series  of  prompts  such  as  "TURN  PRINTER  ON,  ALIGN  PAPER, 
PRESS  ENTER  WHEN  READY"  leads  the  user  through  the  necessary 
steps  in  printing  the  lessons.  Upon  completion  of  printing, 
the  user  is  presented  a  menu  offering  the  choice  of 
returning  to  the  main  lessons-learned  menu  or  exiting  the 
system  and  logging  off.  Once  the  user  completes  the  session 
and  logs  off,  a  final  prompt  showing  total  estimated  session 
cost,  based  on  computer  CPU  time,  appears  on  the  screen 
(NALL  User’s  Manual,  1989:  62-65). 

The  Navy  does  not  bill  NALL  system  users  for  the  CPU 
time  costs  associated  with  system  usage.  The  Navy  firmly 
believes  "all  system  usage  costs  are  saved  many  times  over 
by  the  prevention  of  costly  errors  and  the  improved 
reliability  and  maintainability"  achieved  through  the 
application  of  lessons-learned  IGrimsley,  1989). 

System  Benefits.  The  implementation  and  use  of 
the  NALL  system  has  saved  the  Navy  millions  of  dollars  and 
prevented  an  indeterminable  amount  of  lost  time  due  to 
accidents  and  injuries  (Gardiner  and  Grimsley,  1989). 
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Several  examples  demonstrating  the  usefulness  of  the  NALL 
were  provided  by  Grimsley.  Two  of  those  examples  will  be 
examined  here . 

The  first  example  applied  to  the  fuel  management  panels 
within  the  cockpit  of  the  F-14  aircraft  Navy-wide.  The 
existing  fuel  management  panels  were  under  contract  to  be 
replaced  with  liquid  crystal  display  (LCD)  type  panels  which 
consume  less  power.  The  new  LCD  panels  met  all  required 
specif ications  and  the  conversion  seemed  imminent.  However, 
based  on  a  lesson  learned  in  the  NALL  concerning  LCD 
displays,  a  new  LCD  fuel  management  panel  was  installed  in 
an  F-14  and  subjected  to  lighting  similar  to  operational 
conditions.  The  NALL-inspired  field  test  proved  the  LCD 
panels  would  no.  function  in  the  F-14  as  desired  and  the 
contract  for  conversion  was  cancelled  before  substantial 
costs  were  incurred  (Grimsley,  1989). 

The  second  example  of  a  NALL  lesson  benefitting  the 
Navy  concerned  human  lives.  NALL  researchers  monitoring 
mishap  reports  for  potential  lessons-learned  noted  a 
disturbing  trend.  Aircrew  members  were  suffering  helmet 
loss  during  62.5  percent  of  ejections  occurring  over  300 
knots.  This  trend  coupled  with  other  factors  led  to  the 
identification  of  serious  flaws  in  redesigned  aircrew 
helmets.  The  new  helmets  had  been  redesigned  with  less 
material  in  the  lower  rear  portion  to  increase 
maneuverability;  unfortunately,  this  change  also  increased 
the  probability  of  helmet  loss  and  resultant  aircrew  injury 
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during  ejections.  The  older  helmets  had  a  loss  rate  of  less 
than  23  percent.  Because  of  the  NALL,  the  helmet  loss 
problem  has  been  identified  and  corrected  (Grimsley,  1989). 

NALL  personnel  related  several  other  examples  of  NALL 
lessons  being  utilized  to  reduce  costs,  increase  reliability 
and  maintainability,  improve  acquisition  practices,  increase 
quality  and  improve  safety  and  effectiveness  in  various 
aviation  systems  (Grimsley,  1989). 

NALL  Bulletin  Board.  To  further  increase  the  NALL 
system’s  user-friendliness  and  reduce  the  NALL  system’s 
mainframe  CPO-time  costs,  the  Navy  has  recently  brought  on¬ 
line  the  Naval  Aviation  Lessons-Learned  Remote  Bulletin 
Board  System  (RBBS)  (Blankenship,  1989).  As  stated  in  the 
RBBS  welcome  message,  "The  Lessons-Learned  Remote  Bulletin 
Board  System  is  for  the  dissemination  of  information  from 
the  Naval  Aviation  Lessons-Learned  Program"  (Grimsley, 

1989) . 

The  RBBS  was  initiated  for  several  reasons.  One  of  the 
principle  motivators  of  the  RBBS  was  the  recognition  by  the 
NALL  system  operators  that  certain  types  of  lessons  were 
pulled  from  the  NALL  main-frame  database  far  more  frequently 
than  others.  The  advantage  of  a  PC-based  RBBS  containing 
"hot"  lessons-learned  reports  is  reduced  cost.  The  NALL 
operators  realized  the  “hot"  topic  lessons-learned  reports 
could  be  retrieved  from  the  main-frame  once  (with  associated 
CPD-costs)  and  loaded  onto  a  personal  computer  (PC)  for 
repeated  access  "free”  through  a  bulletin  board  system. 
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For  all  practical  purposes,  the  cost  of  operating  a  PC  RBBS 
is  nothing  (Grimsley,  1989).  The  CPU  costs  of  the  Amdahl 
main-frame  are  currently  in  excess  of  $1500  per  hour.  (NALL 
User’s  Manual,  1989:  65).  Additionally,  because  the 
lessons -learned  packages  loaded  onto  the  RBBS  are  archived, 
i.e.  compressed  electronically,  the  user’s  download  time  is 
reduced  by  better  than  90  percent,  which  decreases  the 
user’s  long  distance  phone  costs  (Grimsley,  1989). 

The  NALL  RBBS  also  offers  another  avenue  for  user  input 
of  potential  lessons  into  the  NALL  system.  Potential 
lessons  can  be  uploaded  to  the  RBBS  where  they  will  be 
forwarded  to  the  research  staff.  Users  can  also  directly 
communicate  with  the  NALL  operators  via  the  message  and  help 
options  built  into  the  RBBS  (Blankenship,  1989). 

In  addition  to  cost  benefits,  the  NALL  RBBS  is  even 
more  user-friendly  than  the  NALL  system  because  it  is 
structured  similarly  to  many  of  the  more  popular  PC  bulletin 
boards  in  use  today.  The  NALL  RBBS  is  a  menu  driven 
bulletin  board  built  using  RBBS  PC  software,  version  16.1 
(Grimsley,  1989).  The  RBBS  resides  on  an  IBM-XT  personal 
computer  in  the  main  office  of  the  NALL  system  operators. 

The  NALL  system  operators  configured  the  menus  of  the  RBBS 
to  be  as  simple  and  self  explanatory  as  possible  (Grimsley, 
1989).  Although  still  in  its  infancy,  the  NALL  RBBS  has 
proven  very  popular  with  the  NALL  users.  The  opening  menu 
of  the  NALL  RBBS,  with  the  options  as  shown  in  Figure  11, 
provides  an  indication  of  the  RBBS  capabilities. 
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COMMUNICATIONS 


PERSONAL 

E)nter  a  message 
K)ill  a  message 
P)ersonal  mail  check 

R) ead  message(s) 

S) can  msgs  (to/from) 

T) opic  scan  messages 


X —  Turns  off  this  display 
(only  bottom  prompt) 


CONFERENCING 
J)oin  conference 


UTILITIES 


H)elp. . .Are  you  lost? 
X ) pert  mode  on/off 
?)List  of  functions 


SYSTEM 

A) nswer  Questionnaire 

B) ulletins  (list,  view) 

C Comments (for  SYSOP  only) 
I)nitial  welcome  message 
0 Operator  (page  SYSOP) 
W)ho  else  is  on  system? 


ELSEWHERE 


D)oors  Subsystem 

F) ile  Subsystem 

G) ood  bye  (hangup) 
Q)uit  to  a  Subsystem 
U)tilities  Subsystem 


Select  a  Command. . . 

MAIN  commands  <?,A,B,C,E,F,H, I,P,Q,R,S,T,U> . . . 


Figure  11.  NALL  Bulletin  Board  Main  Menu 
(NALL  RBBS,  1989) 


Logging  onto  the  NALL  RBBS  requires  only  a  personal 
computer,  modem  and  the  RBBS  phone  number.  Unlike  the  NALL, 
access  to  the  RBBS  does  not  require  prior  approval  from  the 
Naval  Air  Test  Center;  however,  prior  approval  must  be 
obtained  before  uploading  and  downloading  is  allowed.  New 
users  are  limited  to  reading  the  bulletins,  answering 
questionnaires  and  reviewing  the  RBBS  directories 
(Blankenship,  1989). 

Users  of  the  RBBS  have  access  to  the  "hot"  topic 
lessons  discussed  previously.  Alternately,  through  the 
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RBBS,  users  may  leave  requests  for  the  system  operator 
(SYSOP)  to  search  the  main-frame  NALL  system  for  non- "hot" 
lessons  meeting  certain  criteria.  The  SYSOP  performs  the 
lesson  retrieval  from  the  NALL  and  loads  the  lessons/reports 
onto  the  RBBS  system  for  later  downloading  by  the  requesting 
user  (Grimsley,  1989). 

Recommendations .  The  NALL  developers  provided 
several  in-sights  into  what  makes  an  on-line  lessons- learned 
system  such  as  the  NALL  system  successful .  The 
recommendations  noted  follow: 

1.  DEFINE  USERS  INTERFACE  REQUIREMENTS.  This  should 
be  done  down  to  the  smallest  detail,  such  as  how  the  user 
would  like  the  individual  menus  to  appear  and  which  keys 
will  perform  which  functions. 

2.  DATA  REQUIREMENTS.  As  far  as  the  system  programmer 
is  concerned,  "data  is  data".  The  user  must  convey  to  the 
programmer  how  the  data  is  used  in  day-to-day  operations. 
Information  such  as  a)  how  the  data  is  normally  presented  or 
viewed,  b)  how  the  data  is  received  or  input,  c)  how  the 
data  is  manipulated,  d)  how  the  data  is  sorted  and  e)  how 
the  user  wants  to  retrieve  the  data  -  must  be  fully 
explained  to  the  programmer . 

3.  USER-FRIENDLINESS.  Any  system  that  will  be  used 
must  be  user-friendly.  User-friendliness  features  —  such 
as  extensive  built-in  help  screens,  clear  menus,  self- 
explanatory  individual  menu  options,  etc.  —  all  increase 


user-friendliness.  Conversely,  a  means  of  disabling  the 


menus  and  choosing  command  driven  options  should  also  be 
incorporated  so  that  the  experienced  user  can  avoid  the 
menus  if  desired.  The  user  must  always  be  able  to  back  out 
of  a  incorrect  entry  by  hitting  an  “undo”  key  ( Grimsley , 
1989).  This  prevents  the  novice  from  getting  stuck  at  a 
menu  level  and  not  being  able  to  get  back  out. 

4.  DOCUMENTATION.  Documentation  is  also  critical  to 
user-friendliness.  New  users  of  the  NALL  are  provided  an 
extensive,  well-written,  112-page  User’s  Manual.  The  User’s 
Manual  provides  a  step  by  step  explanation  of  how  to  log 
onto  the  system,  access  the  database,  perform  lessons- 
learned  searches,  generate  reports,  generate  retrievals, 
print  lessons  and  reports,  exit  the  database  and  how  to 
reconnect  to  the  system  if  connection  is  inter rupted .  Also 
provided  in  the  User’s  Manual  are  keyboard  mapping  diagrams, 
hardware  and  software  configuration  directions  and 
explanations  on  how  to  alter  the  configurations  to  match  the 
user’s  system.  The  User’s  Manual  has  ten  appendices  further 
explaining  the  NALL.  The  User’s  Manual  Appendix  1  provides 
a  sample  format  letter  for  requesting  access  to  the  NALL. 
Appendices  2  through  4  of  the  User’s  Manual  list  the 
acceptable  impact  areas ,  work  unit  codes  and  aircraft  type 
codes.  Finally,  the  User’s  Manual  Appendices  5  through  10 
provide  program  code  listings  for  the  various  database 
programs  used  on  the  NALL  to  retrieve  lessons,  print 
lessons,  etc.  The  program  code  listings  are  included  in  the 
User’s  Manual  for  reference  purposes  only. 
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5.  CROSSFEED  AND  COMMUNICATE.  The  users,  owners  and 
programmers  must  continually  communicate  to  ensure  the 
expectations  and  efforts  of  each  individual  are  known  in 
advance.  Bi-weekly  meetings  during  the  development  stage 
are  extremely  beneficial  in  preventing  wasted  effort  on  the 
part  of  the  programmers  and  unmet  expectations  on  the  part 
of  the  user  (Dove  and  Grimsley,  1989). 

6.  ADVERTISE.  Once  a  lessons-learned  system  is  on¬ 
line,  it  is  critical  that  its  potential  users  know  of  its 
existence.  Lessons-learned  are  not  static  entities,  but 
continually  evolve  and  change  with  technology,  personnel, 
and  management  changes.  The  NALL  team  has  visited  hundreds 
of  Naval  Stations,  contractor  facilities  and  other  sites, 
always  seeking  new  users  and  new  potential  lessons 
(Grimsley,  1989). 

NALL  Summary .  The  Naval  Aviation  Lessons-Learned 
system  is  one  of  the  largest  and  most  user-friendly  lessons- 
learned  system  found  in  the  DOD.  Over  90  percent  of  the 
feedback  received  from  the  NALL’s  users  indicates  it  is 
"extremely  helpful,  well  thought  out  and  easy  to  use" 
(Grimsley,  1989).  The  development  of  the  NALL  system 
followed  a  " quasi -prototype ,  quasi-systems  analysis" 
methodology  (Dove,  1989).  As  a  result  of  the  NALL’s 
success,  both  the  Army  and  Air  Force  have  visited  the  Naval 
Aviation  Lessons-Learned  center  to  gain  insight  into  how  to 
configure  their  respective  service’s  lessons-learned 
programs.  Similarly,  several  defense  contractors  have 
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initiated  lessons- learned  systems  modelled  after  the  NALL  or 
had  the  NALL  program  installed  at  their  sites  by  the  NALL 
team  (Grimsley,  1989). 

Air  Force  Lessons-Learned  On-Line  Access.  The  0 . S .  Air 
Force  has  also  recognized  that  “lessons-learned  can  and  do 
impact  system  acquisition"  (Kerr,  1989).  Consequently,  the 
Acquisition  Logistics  Division  (ALD)  at  Wright  Patterson 
AFB,  Ohio  established  the  Air  Force  Lessons-Learned  Databank 
(AFLL)  in  1977,  and  automated  the  AFLL  in  1978.  AFLL  is  "by 
far,  the  largest  and  most  mature  of  the  various  ’ Lessons- 
Learaed'  (corporate  memory)  data  banks  now  in  operation" 
(On-Line  Access,  undated:  6).  The  principle  goal  of  the 
AFLL  is  to, 

Provide  feedback  for  improving  all  aspects  of  Air 
Force  Operations.  Application  of  lessons-learned 
is  the  bottom  line.  If  they  are  to  be  applied, 
they  mus~  be  communicated  to  the  decision  makers 
in  current  programs.  (On-Line  Access,  undated:  6) 

History.  The  Acquisition  Logistics  Division  was 
originally  formed  to  help  bridge  the  gap  between  the 
acquisition  and  logistics  communities  to  improve  the 
reliability  and  supportability  of  new  weapons  systems  coming 
into  the  Air  Force  inventory.  To  help  meet  this  goal,  ALD 
initiated  the  Air  Force  Lessons -Learned  Program  (Keith, 

1989).  The  AFLL  system  is  included  in  the  tri-service  Joint 
Agreement  on  the  Joint  Logistics  Commander’s  Lessons-Learned 
discussed  in  the  Naval  Aviation  Lessons-Learned  section.  As 
defined  by  ALD,  a  lesson  learned  is  a  "recorded  experience, 
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that  can  be  of  value  in  the  conduct  of  future  programs"  (On- 
Line  Access,  undated:  6). 

The  initial  AFLL  lessons-learned  were  collected  by 
lessons -learned  teams.  These  teams  conducted  field  trips  to 
a  variety  of  sources  including  private  industry.  Air 
Logistics  Centers,  AF  Safety  Center,  MAJCOMS,  Air  Force 
Systems  Command  Product  Divisions,  conferences/meetings  and 
DOD  agencies.  The  AFLL  system  has  377  on-line  access  users, 
however  many  "users"  are  in  fact  organizations  which  may 
have  30  or  40  personnel  sharing  one  user-id  and  password 
(Kerr,  1989). 

System  Configuration.  The  AFLL  database  is  hosted 
on  Aeronautical  Systems  Divisions’  Information  Central 
Division  System  1  (VAX  11/780),  minicomputer  located  at 
Wright-Patterson  AFB,  OH  (On-Line  Access,  undated:  6). 
Batelle’s  Automatic  Searching  and  Indexing  System  (BASIS) 
controls  the  actual  lessons-learned  data  manipulation 
(Keith,  1989).  The  AFLL  can  be  accessed  through  any 
terminal  directly  tied  to  the  VAX  mainframe.  Alternatively 
the  AFLL  is  accessible  to  remote  personal  computer  users 
possessing  a  user-id  and  password  provided  by  the  ALD  upon 
request.  Remote  users  must  have  a  personal  computer,  a 
modem  and  a  communications  package  capable  of  emulating  a 
VT100  terminal.  Other  terminal  emulators  will  work,  but  a 
VT100  emulator  works  best.  Finally,  the  AFLL  is  accessible 
through  the  Defense  Data  Network  (DDN)  (On-Line  Access, 
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undated:  12).  The  AFLL  is  available  for  access  24  hours  per 
day,  seven  days  per  week  (Keith,  1989). 

AFLL  Database  Configuration.  The  AFLL  database 
currently  contains  over  2000  technical  and  management 
iessons-learned  (Purvis,  1989).  As  outlined  in  the  AFLL  On- 
Line  Access  User's  Guide,  each  lesson  contains  the  six 
different  fields  shown  below: 

1.  CALL  NUMBER:  An  office  assigned,  sequential 
number,  by  which  each  unique  lesson  can  be  identified  and 
retrieved. 

2.  TOPIC:  Brief  but  representative  description  of  the 
content  of  the  lesson. 

3.  LESSON  LEARNED:  One  or  two  concise  sentences, 
showing  a  cause  and  effect  relationship,  and  stating  the 
single  most  important  finding. 

6.  PROBLEM:  Normally  no  longer  than  one  or  two  brief 
sentences,  this  field  will  say  "none"  for  positive  Iessons- 
learned. 

7.  DISCUSSION:  One  to  three  paragraphs  giving  as 
complete  an  account  of  the  situation  as  possible. 

8.  APPROPRIATE  ACTION:  This  is  the  most  important 
part  of  a  lesson  learned.  It  will  detail  who  should 
accomplish,  what  task,  when  in  the  acquisition  cycle  to 
apply  the  knowledge  previously  gained  in  that  situation . 
(On-Line  Access,  undated:  9). 

The  format  of  the  AFLL  Iessons-learned  is  very  similar 
to  the  format  of  the  Navy’s  NALL  Iessons-learned.  The  Air 
Force  and  Navy  Iessons-learned  offices  frequently  share 
information  and  ideas  to  improve  both  systems  (Grimsley, 

1989;  Kerr,  1989).  A  transcription  of  a  sample  lesson 
retrieved  from  the  AFLL  is  shown  in  Figure  12. 

Data  Input.  Potential  Iessons-learned  for  the  AFLL 
are  obtained  from  a  variety  of  sources  including  successful 
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CALL  NUMBER:  0728 
TOPIC:  ENGINE  STARTING 

LESSON  LEARNED:  The  use  of  gear  tooth  coupler  pawls  for 
dynamic  engagement  in  the  engine  starting  cycle  should  be 
avoided  unless  torsion  surges  can  be  accommodated  through 
a  torsion  shaft  fluid  coupling,  or  flat  disk  clutch. 

PROBLEM:  Excessive  maintenance  manhours  and  high  level 

of  force  degradation  have  resulted  from  the  engine 
starting  engagement  system  design  using  pawls. 

DISCUSSION:  The  high  performance  fighter  aircraft 

central  gearbox  (CGB)  engages  an  airframe  mounted 
accessory  gear  box  through  a  gear- type  pawl  to  transmit 
engine  starting  torque  from  the  jet  fuel  starter  to  the 
engine.  The  shock  loading  of  the  pawls  during 
ratcheting/ grinding  engagement  destroys  the  pawls  and 
creates  a  large  shear  stress  on  the  coupling  stubs. 

The  high  performance  fighter  aircraft  force  degradation 
contribution  listing  ranked  the  CGB  (WUC  24ANO)  as  eighth 
and  the  coupling  stub  (WUC  24ANH)  as  twenty -eighth.  Zero 
RPM  engagement  parameters  are  not  compatible  with 
operational  reality  of  consecutive  start  attempts  without 
attaining  zero  RPM  between  attempts. 


APPROPRIATE  ACTION:  Future  engine  starting  subsystems 
should  be  designed  to  avoid  dynamic  pawl/gear  engagement 
and  minimize  shock  loading  of  the  system  with  shear 
stresses.  The  use  of  air  turbine  motors,  electric  motors, 
fluid  couplings  or  slip  disk  clutching  as  alternatives  to 
hard  engagement  should  be  considered. 


Figure  12.  Sample  AFLL  Lesson 
(AFLL,  1989) 


or  unsuccessful  acquisition  program  experiences,  personal 
experiences,  test  reports,  inspection  deficiencies, 
maintenance  data,  engineering  data  and  contractors.  Also, 
all  users  of  the  AFLL  are  encouraged  to  submit  potential 
lessons  to  ALD  by  the  AFLL  opening  message  of  the  AFLL: 
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In  order  to  stay  abreast  of  changing  technology 
affecting  weapon  system  development  and  maintain  a 
dynamic  data  base  the  Air  Force  lessons- learned 
program  relies  on  feedback  from  each  organization 
(DOD  or  contractor)  that  designs,  acquires, 
operates,  or  supports  an  Air  Force  system.  To 
achieve  this  goal  we  welcome  potential  lessons- 
learaed  submittals  from  anywhere  and  anyone .  One 
of  the  options  on  the  following  main  menu  screen 
has  been  provided  for  you  to  pass  potential 
lessons  directly  to  the  Data  Bank  staff  for 
validation.  (AFLL,  1989) 

Potential  lessons-learned  are  submitted  on  an  Air  Force 
Form  1251  to  ALD  Directorate  of  Lessons -Learned  and  Systems 
Support  (ALD/LSL)  for  research  and  validation.  The 
potential  lesson  learned  is  then  reviewed  for  validity  by 
one  of  several  "functional  experts"  in  ALD.  A  potential 
lesson  is  considered  valid  if  it  is  “ technically  accurate 
and  not  contrary  to  established  regulations"  (Keith,  1989). 
On like  the  Navy  Lessons-Learned  database  which  has  an  in- 
house  staff  of  researchers  whose  primary  duty  is  lessons- 
learned  research  and  validation,  the  AFLL  validates 
potential  lessons  with  the  help  of  functional  experts 
operating  in  an  "additional  duty”  capacity  (Keith,  1989). 
ALD/LSL  maintains  of  a  file  of  over  2,000  “no  lessons- 
learned"  obtained  from  lessons  that  have  been  overcome  by 
events,  or  submitt.  13  that  could  not  be  validated  (Kerr, 

1989) . 

Upon  completion  of  the  validation,  the  researcher 
conducts  a  format  review  of  the  lesson  for  compliance  with 
the  "cause  and  effect"  relationship  and  "who,  what,  when" 
format  desired  in  all  AFLL  lessons  (Keith,  1989).  The  cause 


74 


and  effect  relationship  explains  "if  an  action  is/is  not 
accomplished  what  event  will/will  not  occur"  and  the  who, 
what,  when  explains  "in  detail  who  should  accomplish  what 
task,  when  in  the  acquisition  cycle  to  apply  the  lesson" 
(On-Line  Access,  undated:  8).  After  the  format  review  and 
validation  of  the  lesson  by  the  functional  expert,  the 
approved  lesson  is  signed  off  by  the  Director,  ALD/LSL  and 
entered  into  the  AFLL  database  (Purvis,  1989). 

The  Air  Force  follows  the  same  policy  as  the  Navy 
regarding  sanitizing  of  lessons-leamed,  for  the  same 
reasons,  i.e.  all  lessons -learned  will  be  sanitized  such 
that  neither  the  lesson  nor  the  backup  documentation 
attached  contains  any  reference  to  the  person,  unit, 
organization  or  firm  associated  with  the  lesson  (Keith, 

1989) . 

Data  Retrieval .  Like  the  Naval  Aviation  Lessons- 
Learned  system,  the  AFLL  is  also  menu-driven  allowing  the 
user  to  select  which  activity  is  desired  from  a  preset 
"menu"  of  options.  Each  menu  is  self-explanatory  and  the 
options  available  are  clearly  shown.  The  opening  menu  of 
the  AFLL  system  allows  the  user  to  select  from  a  variety  of 
options  including  searching  for  lessons,  submitting 
potential  lessons,  and  others  as  shown  in  Figure  13.  Rather 
than  pressing  a  number  associated  with  a  particular  menu 
option,  the  AFLL  menus  are  configured  to  allow  option 
selection  by  .  ressing  the  first  letter  associated  with  the 
menu  option  desired  (AFLL,  1989). 
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U.S.A.P.  Lessons-Leamed  DATA  BANK  **MAIN**  MEND 
Line  number  1  with  2131  lessons. 

Please  select  by  LETTER  from  the  options  listed  below. 
S.  (S)earch/Retrieve  specific  lessons. 

P.  submit  (P)otential  lessons  to  the  data  bank. 

C.  leave  (C)oaments/Suggestions  for  the  data  bank  staff. 
0.  go  to  (D)tilities  menu. 

L.  change  menu  drive  (L)evel  (amount  of  assistance 
provided ) . 

Q.  (Q)uit  the  data  base. 

(S)earch, (P)otential, (C)omment, (O)til. , (L)evel, (Q)uit?= 


Figure  13.  AFLL  Main  Menu  (AFLL,  1989) 

Lessons  input  into  the  AFLL  are  indexed  into  categories 

with  the  use  of  logistics  elements  and  management  elements. 

Management  Lessons  address  program  decisions  and 
actions  in  such  areas  as  program  control, 
budget/financial  control,  contracting  techniques, 
support  planning,  configuration  management, 
maintenance  concepts  and  data  management. 

Technical  Lessons  relate  to  systems,  equipment  and 
components,  including  hardware,  software,  support 
equipment,  or  the  design  factors  that  influence 
the  performance  of  the  system  or  equipment. 

(Keith,  1988:  atch  2) 

This  allows  users  to  conduct  narrower  keyword  searches  in  a 
particular  area,  without  retrieving  lessons  containing  the 
keyword  but  no  applicable  data.  Conversely,  all  lessons 
within  an  area  can  be  retrieved  without  keyword  searches. 
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The  management  and  logistics  elements  of  the  AFLL  are  shown 
here: 


i  LOGISTICS  ELEMENT  AREAS 


MANAGEMENT  ELEMENT  AREAS 


1  COMPUTER  RESOURCES  (SUPPORT) 

2  ENERGY  MANAGEMENT 

3  ENGINEERING  DATA  (TECH.  DATA) 

4  FACILITIES 

5  FUNDING  (LOGISTICS  SUPPORT) 

6  LOGISTICS  MGT.  INFO.  SUPPORT 

7  MAINTAINABILITY 

8  MAINTENANCE  CONCEPT  (PLANNING) 

9  MODIFICATION  PLANNING 

10  MANPOWER  REQUIREMENTS 

11  RELIABILITY 

12  RELIABILITY  &  MAINTAINABILITY 

13  SAFETY 

14  SUPPLY  SUPPORT 

15  SUPPORT  EQUIPMENT 

16  SURVIVABILITY 

17  TECHNICAL  ORDERS  (TECH.  DATA) 

18  TEST  AND  EVALUATION 

19  TRANS.  PACKAGING  &  HANDLING 

20  TRAINING  AND  TRAINING  SUPPORT 

21  ARTIFICIAL  INTELLIGENCE 


30  CONFIGURATION  MANAGEMENT 

31  CONTRACT  ADMINISTRATION 

32  CONTRACTING 

33  DATA  MANAGEMENT 

34  ENGINEERING 

35  FOREIGN  MILITARY  SALES 

36  HUMAN  FACTORS  ENG. 

37  LIFE  CYCLE  COST 

38  MANUFACTURING 

39  OPERATIONAL  REQUIREMENTS 

40  PROGRAM  CONTROL 

41  QUALITY  ASSURANCE 

42  SOURCE  SELECTION 

43  PROG.  MGT.  TRANSFER 

44  LOG.  SUPPORT  ANALYSIS 

45  PROGRAM  MANAGEMENT 

46  ENVIRONMENTAL  MANAGEMENT 

47  WARRANTIES 


(AFLL,  1989). 


Data  Sorting  and  Retrieval.  If  the  user  selects 
the  " (S)earch/Retrieve  specific  lessons"  menu  option,  a 
submenu  is  presented  offering  the  user  a  selection  of 
various  search  and  retrieval  methods.  The  (S)earch/Retrieve 


submenu  is  shown  in  Figure  14.  The  flexibility  of  the  AFLL 


Search/Retrieve  options  allows  the  user  to  retrieve  lessons 
many  different  ways.  Wordsearch  retrievals,  either  of  the 


entire  file  or  only  a  particular  log  or  management  element, 


search  based  on  an  "and"  type  search,  which  requires  all  of 


the  key  words  to  be  found  in  the  lesson(s)  before  the 
lesson(s)  «ili  be  retrieved  While  the  AFLL  system  supports 
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Lessons -Learned  **SEARCH/RETRIEVE**  MEND  Pg.  1 
Line  number  1  with  2140  lessons. 

Please  select  by  LETTER  from  the  options  listed  below. 

W.  (W)ordsearch  the  entire  active  Lessons-Learned  file. 

L.  keyword  search  for  Lessons  from  a  particular  (L)0G 
element  area  only;  (this  may  provide  a  faster  more 
accurate  search ) . 

N.  search  for  specific  Lessons  by  Lessons  (N)umber. 

E.  Retrieve  all  of  the  Lessons  for  a  particular  LOG 
(E)lement . 

M.  Retrieve  lessons  uploaded  or  (M)odified  within  last 
30  days . 

V.  (V)iew  more  search  options. 

Q.  (Q)uit  the  data  base. 

(W)ordsearch,  (L)0G  Keyword,  (N)umber,  (E)lement, 

( M ) od ,  (V)iew,  (Q)uit?==> 

Figure  14.  AFLL  Search/Retrieve  Menu 
(AFLL,  1989) 

multiple  wordsearching,  (up  to  6  words  or  58  characters)  it 
does  not  accomplish  “phrase"  searching,  i.e.  the  words  in 
the  search  string  will  be  contained  somewhere  in  the  lessons 
retrieved,  but  may  not  be  in  the  exact  sequence  of  the 
search  string  (On-Line  Access,  undated:  19).  The  further 
options  of  the  sort  and  display /print  menus  are  shown  in 
Figures  15  and  16.  Upon  search  completion,  the  AFLL  advises 
the  user  how  many  lessons  have  been  found  and  prompts  the 
us  .r  with  several  options  including  listing  the  lesson 
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Lessons-Learned  SORT  MENU 
Please  select  by  LETTER  from  the  options  below. 
SORT  BY: 

N.  Lesson  learned  (N)umber  (199). 

A.  (A)lphabetically  by  topic. 

R.  Lesson  currency  (R)eview  date. 

D.  go  to  (D)isplay /Print  menu. 

(N)umber,  (A)lphabetically, ,  (R)eview, 
(D)isplay,  (Q)uit?==> 

Figure  15.  AFLL  Sort  Menu  (AFLL,  1989) 


Lessons-Learned  DISPLAY/PRINT  MEND 

Please  select  by  LETTER  from  the  options  below: 

D.  (D)isplay  Lessons  (or  print  locally)  in 
Lesson  Learned  format. 

A.  (A)bstract  format;  Number,  Topic,  and 
Lesson  Learned. 

B.  (B)rief  format.  Lesson  Learned,  Problem, 
and  Appropriate  Action. 

L.  (L)ist  the  Topics  and  LL  Numbers. 

P.  (P)rint  Lessons  (500  max)  at  ASD, 

Wright -Patterson  AFB,  Ohio. 

S.  Change  the  document  set  (S)ort  order. 

Q.  (Q)uit  the  data  base. 

NOTE:  Select  DISPLAY  for  local  prints— PRINT 

sends  lists  to  ASD,  WPAFB,  Ohio. 

(D)isplay,  (A)bstract,  (B)rief,  (L)ist, 

(P)rint,  (S)ort,  (Q)uit?==> 

Figure  16.  AFLL  Display/Print  Menu  (AFLL,  1989) 
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topics  and  numbers,  conducting  another  wordsearch,  quitting 
the  database  or  entering  the  sort  or  display /print  menus. 

Printing  Lessons-Learaed.  Hard-copies  of  the 
lessons  retrieved  from  the  AFLL  system  can  be  printed  either 
at  the  user’s  remote  location  (through  the  (D)isplay  option 
of  the  Print/display  menu)  or  at  the  host  computer  printer 
at  Wright  Patterson.  There  is  normally  a  one  day  turnaround 
on  lesson  printing  at  the  host  computer  (On-Line  Access, 
undated:  19).  Alternatively,  users  may  make  mail  or  phone 
requests  to  ALD/LSL  for  hardcopy  lessons- learned 
information,  abstracts  (brief  synopsis  of  all  lessons), 
bulletins  (lessons  on  specific  topics  such  as  composites, 
engines,  quality  assurance,  etc.)  or  lesson  packages 
tailored  to  the  user’s  needs  (On-Line  Access,  undated:  6). 

Four  distinct  lesson  formats  can  be  obtained  from  the 
AFLL  including  1)  Lesson  Learned  Format  which  displays  all 
information;  2)  Abstract  Format  which  displays  only  the  call 
number,  topic,  and  lesson  learned  statement;  3)  Brief  Format 
which  displays  all  information  except  the  discussion;  and 
finally,  4)  Topics  and  Numbers  Format  which  displays  only 
the  call  numbers  and  lesson  topics  in  tabular  format  (On- 
Line  Access,  undated:  8). 

The  Air  Force  does  not  bill  AFLL  system  users  for  the 
CPU  time  costs  associated  with  system  usage  for  the  same 
reasons  the  Navy  does  not  bill  NALL  system  users  -  system 
usage  costs  are  saved  by  the  prevention  of  costly  mistakes 
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and  improved  reliability  and  maintainability  achieved 
through  the  application  of  lessons- learned  (Keith,  1989). 

System  Benefits.  The  implementation  and  use  of 
the  AFLL  system  has  yielded  substantial  benefits  to  the  Air 
Force.  However,  quantification  of  these  benefits  is 
extremely  difficult.  One  indication  of  the  success  of  the 
AFLL  is  the  current  interest  of  ALD/LSL  in  expanding  the 
AFLL's  capabilities  to  include,  among  others  -  graphics, 
proximity  searching,  spell  checking  and  cut  and  paste 
functions.  Another  indication  of  AFLL  success  is  system 
usage  -  approximately  600  new  potential  lessons  are  received 
yearly  by  ALD/LSL  for  research  and  validation  (Kerr,  1989). 

p^n^i^ndations .  The  AFLL  developers  provided 
several  in-sights  into  what  makes  on-line  lessons- learned 
systems  successful.  The  recommendations  noted  were  similar 
to  the  NALL  developer’s  recommendations  and  also  included: 

1)  ADVERTISE.  Let  the  people  know  who  you  are. 

ALD/LSL  advertises  through  the  use  of  pamphlets,  base 
newspaper  articles  and  briefings  to  other  organizations . 
ALD/LSL  has  promoted  lessons- learned  at  the  Air  Force 
Institute  of  Technology  and  the  Senior  NCO  Academy.  In 
essence,  if  the  user  isn’t  aware  of  the  lessons-learned 
program,  the  program  can  not  be  successful  (Kerr,  1989). 

2)  VALIDATE.  Lesson  validation  is  critical.  If  users 
conduct  searches  only  to  discover  lessons  that  are 
technically  incorrect,  counter  to  established  regulations, 
outdated  or  otherwise  inappropriate  they  will  very  soon  quit 
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using  the  lessons-learned  system.  Validation  must  be  a 
continual  process,  where  lessons  are  reviewed  frequently  and 
updated  as  necessary  (Kerr,  1989). 


3)  IT  SIMPLE.  Lessons-Learned  systems  must  be 

simple  to  use.  The  AFLL  operators  consider  this  an  endless 
process.  Osers  are  encouraged  to  advise  ALD/LSL  of  any 
problems  encountered  or  suggestions  that  might  improve  the 
system’s  simplicity.  A  recent  example  of  this,  related  by 
Kerr,  involved  users  having  difficulty  preventing  the 
lessons  from  scrolling  across  the  screen  too  fast.  Although 
only  certain  users  were  experiencing  this  problem,  ALD/LSL 
modified  the  AFLL  to  include  page  breaks  every  20  lines  in 
the  lessons,  thereby  solving  the  problem  (Kerr,  1989). 

AFLL  Summary.  The  Air  Force  Lessons-Learned 
system  is  the  largest  and  oldest  lessons-learned  system 
found  in  the  DOD  (Keith,  1989).  The  lessons-learned  in  the 
AFLL  system  are  weapon  systems  acquisition  oriented  and 
normally  written  in  a  concise  format,  one  typed  page  or 
less,  for  user  convenience  and  readability  (Purvis,  1989). 
The  development  of  the  AFLL  system  did  net  entail  a  full 
systems  analysis,  but  instead  followed  a  quasi -proto type, 
quasi-systems  analysis  methodology  (Kerr,  1989).  The  AFLL 
allows  users  to  search  for,  retrieve  and  print  lessons  in 
many  ways.  As  a  result  of  the  AFLL’s  success,  efforts  are 
now  underway  to  expand  the  capabilities  of  the  AFLL. 


V.  Findings  Related  to  WANG  MIS  Development 

Introduction 

Chapter  V  provides  the  answer  to  the  one  remaining 
research  question  that  has  not  yet  been  addressed.  Namely, 
"From  the  user’s  perspective,  how  should  the  proposed 
lessons-learned  MIS  be  structured? ” .  As  noted  in  the 
methodology,  this  question  was  answered  through  the  use  of 
interviews.  Interview  questions  are  shown  in  Appendix  A. 
Answers  to  the  interview  questions  are  presented  below. 

The  Dser’s  Perspective 

Profile  of  the  User.  The  first  portion  of  the 
interview  established  a  profile  of  the  users  selected  for 
interviewing.  This  profile  is  provided  as  a  baseline  to 
understanding  who  the  users  are,  in  terms  of  experience  and 
background,  both  in  construction  management  and  computers. 

Experience .  The  users  interviewed  were  both 
military  and  civilian  in  the  grades  of  captain,  major,  GS-13 
and  GM-14.  The  users’  experience  in  construction  management 
ranged  from  a  low  of  4  years  to  a  high  of  16  years,  with  an 
average  experience  of  10  years.  All  of  the  interviewees 
have  worked  a  broad  range  of  construction  efforts,  from 
small,  low-dollar,  base-level  construction/ropair  to  large, 
AFRCE-  or  MAJCOM-level ,  multi-million  dollar  new 
construction.  Within  the  context  of  construction 
management,  the  interviewees  have  all  functioned  to  some 
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degree  as  inspector,  designer,  technical  advisor,  design 
manager,  construction  manager,  and  project  manager.  All  of 
those  interviewed  related  that  experience  is  crucial  to 
successful  construction  management. 

In  terms  of  computer  experience,  all  of  those 
interviewed  appeared  to  have  similar  ability  and 
understanding.  All  of  the  interviewees  are  familiar  with 
computers,  and  use  computers  in  their  current  jobs.  The 
interviewees  typically  use  computers  for  project  management, 
scheduling,  word  processing,  database  manipulation  and 
spreadsheet  calculations.  Although  most  familiar  with  MS- 
DOS  or  Macintosh  personal  computers,  the  interviewees  all 
expressed  some  familiarity  with  the  WANG/WIMS  computer 
system.  All  but  one  of  the  interviewees  has  written 
computer  programs  either  in  Fortran,  Basic  or  a  higher- level 
computer  language,  although  none  proclaimed  proficiency  in 
this  area.  One  interviewee  found  the  WANG  computer  system 
somewhat  user-friendly,  while  the  others  considered  the  WANG 
less  than  user-friendly.  Only  two  interviewees  had  heard  of 
the  concept  of  on-line  lessens- learned  systems. 

General  System  Parameters .  When  completely 
unconstrained,  the  interviewees  proposed  several  interesting 
idei.  on  what  the  ideal  on-line  lessons- learned  system  would 
be  like.  Many  envisioned  systems  that  would  be  interactive 
on  both  an  audio  and  visual  basis. 

The  Ideal  System.  The  ideal  system  would  allow 
users  to  store  lessons  in  much  the  same  way  as  pocket-size 
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tape  recorders  work,  by  simply  turning  on  a  small  recorder 
and  explaining  the  lesson.  The  recording  devices  could  be 
taken  to  the  construction  site  and  used  whenever  and 
wherever  a  lesson  was  learned.  In  addition  to  the  voice 
lessons-learned  description,  a  visual  picture  of  the  actual 
problem  and  solution  could  be  captured.  Retrieval  of  the 
lessons  from  the  ideal  system  would  also  be  voice  activated. 
The  lessons-learned  seeker  could  audibly  request  all  lessons 
that  meet  certain  criteria.  Upon  completion  of  the  data 
search,  the  system  would  audibly  prompt  the  user  with  a 
brief  synopsis  of  the  lessons  found.  If  a  lesson  synopsis 
appeared  applicable,  the  user  could  then  audibly  request 
additional  information  with  full  video  complements.  Other 
variations  on  this  theme  were  also  postulated  by  the 
interviewees . 

Realistic  Systems.  All  of  the  interviewees 
acknowledged  that  the  proposed  ideal  system  could  not  be 
practically  expected  in  the  near  future.  In  lieu  of  the 
ideal  system,  the  interviewees  provided  several  ideas  of 
what  a  capable  lessons-learned  system  would  be,  within  the 
financial,  technical  and  political  reach  of  Civil 
Engj  neering. 

The  users  interviewed  provided  three  different  concepts 
for  the  proposed  lessons-learned  system.  One  concept 
envisioned  a  system  containing  lessons  stored  in  much  the 
same  way  as  data  in  an  electronic  encyclopedia .  Lessons 
would  be  stored  according  to  the  type  of  work  --  structural, 
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mechanical,  electrical,  etc.  Users  would  then  have  a  master 
index  of  all  major  work  areas  and  could  seek  out  lessons  by 
retrieving  only  the  desired  area  of  interest. 

For  example,  if  the  user  knew  the  contractor  was  about 
to  begin  the  interior  electrical  work  within  a  facility,  the 
user  could  select  ELECTRICAL  from  the  various  types  of  work. 
This  is  conceptually  similar  to  pulling  one  volume  from  a 
set  of  encyclopedias .  Next ,  the  user  could  select  what 
subarea  within  the  major  area  of  ELECTRICAL  work  was 
appropriate,  from  such  options  as  INTERIOR,  EXTERIOR,  etc. 
This  is  similar  to  turning  to  a  specific  chapter  within  the 
encyclopedia  volume  selected.  Successively,  the  user  then 
proceeds  through  his  selection  down  to  the  actual  lessons. 
The  actual  lessons  may  be  several  menus  down,  for  example  - 
lessons  on  electrical  panelboard  work,  in  main  distribution 
systems,  within  the  interior  of  a  facility  could  be  four 
menus  down  from  the  top  menu. 

The  encyclopedia  style  system  would  be  menu-driven,  and 
the  user  would  not  have  to  enter  any  information,  merely 
select  from  a  choice  of  options.  Each  menu  selection  would 
yield  a  submenu  of  subareas  within  the  menu  one  level  up. 

An  example  of  menus  and  submenus  of  this  type  of  system  is 
shown  in  Figure  17. 

A  second  proposed  concept  for  the  lessons-learned 
system’s  structure  is  very  similar  to  that  used  by  the  NALL 
and  AFLL  systems.  Under  this  concept,  the  user  performs 
searches  of  the  entire  database  with  retrieval  based  on 
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FACILITY  ACQUISITION 
LESSONS  LEARNED 
MAIN  MENU 

[Search/Retrieve  Lessons; 
Input  Lessons 
Print/Sort  Lessons 


<Space>  to  Option  Desired, 
Hit  <Enter>  When  Ready... 


PP-3  Utilities 

PF-4  Shell 

PF-5  Cancel;  PF-fl  Previous 

PF-7  Main  M. 

PF-16  Exit 

Figure  18.  Proposed  Master  Menu 


The  third  proposed  concept  was  a  combination  of  the 
first  two.  Under  the  third  concept,  the  initial  menu  would 
offer  the  user  two  methods  of  system  usage  —  encyclopedia 
style,  or  criteria  specified  search  and  retrieval  style. 

The  interviewees  noted,  that  both  methods  of  system  usage 
could  be  helpful  depending  on  the  type  of  lesson  sought.  On 
those  instances  where  the  area  is  clearly  defined  within  one 
specific  work  discipline  —  the  encyclopedia  style  would 
probably  be  the  fastest.  For  multi-discipline  area  work 
efforts,  the  criteria  specification  method  would  work  best. 

System  Characteristics  and  Functions.  Regardless 


which  of  the  three  concepts  outlined  above  is  operative, 
several  general  systems  parameters  are  equally  important. 


For  example,  the  lessons- learned  system  should  be  on  the 
WANG  computer  system.  The  WANG— is^the  system  used  by  Civil 
Engineering,  and  building  a  lessons-lehrned  on  any  other 
system  would  increase  costs  (for  new  equipment)  and  increase 
training  requirements  time. 

Although  the  interviewees  acknowledged  the  need  to  keep 
the  proposed  system  WANG-based,  at  the  same  time  all 
expressed  interest  in  improving  on  WANG  conventions  where 
possible.  For  example,  rather  than  use  programmed  function 
(PF)  keys  to  select  menu  options,  the  lessons-learned  system 
should  allow  menu  item  selection  by  either  lightbar,  number 
or  first  letter  of  the  item  description.  Most  interviewees 
considered  selection  by  lightbars  preferable  to  PF-key, 
number  or  letter  selection.  A  lightbar  is  a  means  of 
highlighting  a  menu  option  such  that  the  active  option  is 
shown  on  the  screen  in  a  different  color  or  reverse  video. 
The  lightbar  is  depicted  in  Figure  18  (on  the  previous  page) 
as  a  dotted  rectangle  around  the  active  option. 

PF-keys  could  be  reserved  for  functions  that  are 
universal  across  all  menu  levels,  on  every  screen  -  such  as 
Help,  Print  Screen,  Utilities,  Shell  to  System,  Previous 
Menu,  Cancel,  Main  Menu  and  Exit.  The  PF-key  options  are 
shown  at  the  bottom  of  Figure  18  on  the  previous  page.  This 
prevents  unnecessary  confusion  created  when  the  same  PF-keys 
perform  different  functions  at  different  menu  levels. 

The  Help  function  should  provide  drop  down  panels  of 
text  explaining  each  option  on  the  screen  in  a  concise 


89 


format.  Extended  help  text  on  each  option  should  also  be 
available,  but  only  appear  when  requested. 

The  Print  Screen  function  should  allow  the  user  to  send 
whatever  is  showing  on  the  screen  at  the  instant  the  Print 
Screen  key  is  pressed  to  either  a  file  or  printer.  Pressing 
the  Print  Screen  PF-key  should  prompt  the  user  with,  "Send 
Screen  to  (F)ile  or  (P)rinter?"  Once  the  user  selects  "F" 
or  "P” ,  the  Print  Screen  function  should  determine  which 
output  device  to  send  the  screen  to  from  the  defaults  set  by 
the  Utilities  function.  If  (F)ile  is  selected,  the  user 
should  be  prompted  with,  “Filename?"  and  allowed  to  enter  a 
filename  where  the  screen  should  be  saved.  Lastly,  the  user 
should  be  allowed  the  option  of  appending  the  information  to 
an  existing  file. 

The  Utilities  function  i3  envisioned  as  a  means  of 
allowing  the  user  to  change  several  system  defaults  such  as 
terminal  configurations,  input/output  devices  (mouse, 
keyboard,  printer,  monitor,  etc. )  file  transfer  rates,  help 
levels,  menu  levels  (expert/novice),  etc.  The  Utilities 
function  should  allow  the  defaults  to  be  changed  either 
temporarily  (for  th  urrent  session  only)  or  saved  and  used 
as  the  defaults  the  next  time  that  user  enters  the  system. 
The  Utilities  function  should  also  allow  the  user  to  print 
the  full  help  text  document. 

The  Shell  to  System  function  should  provide  shell 
capability  to  keep  the  lessons- learned  system  running,  i.e. 
searching  or  printing  lessons,  yet  allow  the  user  to  enter 
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and  temporarily  run  another  application  on  the  computer 
system.  One  example  of  this  function  provided  by  the 
interviewees  was  if  the  user  was  performing  a  lessons- 
learned  search  or  print  session  and  needed  to  run  another 
application  such  as  Project  Design  and  Construction  (PDO), 
the  system  should  allow  the  search  to  continue  in  the 
background  (invisible  to  the  user)  while  the  PDC  application 
was  running  in  the  foreground  (visible  to  the  user). 

The  Previous  Menu  function  should  allow  the  user  to 
back  out  of  the  menu  layers  one  at  a  time.  The  Cancel 
function  should  ''undo'*  the  last  keystroke  and  return  the 
user  to  the  condition  prior  to  the  keystroke.  The  Main  Menu 
function  should  go  directly  to  the  Main  Menu  regardless 
which  menu  level  is  currently  showing  on  the  screen. 

Finally,  the  Exit  function  should  shut  down  the 
lessons-learned  system,  close  all  files,  etc.  The  Exit 
function  should  prompt  the  user  with  "Exit  Lessons-Learned? 
Y/N  "  for  confirmation  thal  the  exit  is,  in  fact,  desired 
and  not  the  result  of  a  missed  keystroke. 

Access  to  the  lessons-learned  system  should  be  on  a 
full  and  restricted  basis.  Full  access  users  should  be 
allowed  to  add  lessons,  retrieve/print  lessons,  and  perform 
system  configuration  changes  on  all  areas  and  lessons. 

Base,  MAJCOM,  AFRCE  and  Air  Staff  engineers  involved  in  Air 
Force  Construction  should  be  considered  full  access  users. 
Restricted  use  should  be  available  to  those  organizations 
with  indirect  involvement  in  the  Air  Force  construction 


process  such  as  contractors.  Army  COE,  NAVFAC,  and 
contracting  personnel.  Restricted  users  should  only  be 
allowed  to  access  information  that  has  been  sanitized. 

Lesson  Structure .  In  discussing  the  structure  of  the 
actual  lessons-learaed,  the  interviewees  were  remarkably 
consistent  in  their  opinions  and  ideas.  Regardless  whether 
the  encyclopedia  style  system  or  criteria  search  and 
retrieval  system  (or  combination  of  the  two)  is  selected, 
the  length,  content  and  format  of  the  lessons-leamed  as 
described  by  the  users  is  very  similar  to  that  used  by  the 
AFLL  and  NALL. 

Length-  While  there  should  not  be  a  limit  on  the 
length  of  a  lesson  learned,  it  is  expected  a  typical 
construction  management  lesson  learned  will  be  one  to  two 
typed  pages,  single  spaced. 

Content .  Lessons -learned  should  contain  at  least 
the  ten  sections  addressed  below: 

1.  LESSON  NUMBER:  Internal  tracking  number  unique  to 
each  lesson  learned,  assigned  by  AFESC  or  the  system. 

2.  IMPACT  AREAS:  A  listing  of  up  to  six  major  areas 
that  the  lesson  affects.  Examples:  safety,  engineering, 
contracting,  human  factors,  health,  training,  environment, 
reliability,  maintainability,  etc.  The  initial  listing  of 
impact  areas  will  be  generated  by  AFESC  after  a  sufficient 
collection  of  lessons  has  been  obtained. 

3.  TOPIC:  Title/subject  of  the  lesson  learned,  brief 
but  representative  of  the  content  of  the  lesson. 

4.  LESSON  LEARNED:  The  actual  lesson  learned,  its 
cause  and  its  effect. 

5.  PROBLEM:  Descriptive  statement  of  what  went  wrong. 
If  the  lesson  is  positive,  this  area  says  "NONE". 

6.  DISCUSSION:  In-depth  discussion  of  lesson 
specifics  such  as  who,  what,  when  and  where.  This  portion 
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of  the  lesson  is  the  unsanitized  portion  —  used  only  by 
full  access  users  who  need  to  locate  the  personnel  involved. 


7.  APPROPRIATE  ACTION:  Recommendation (s )  on  possible 
ways  to  avoid  the  problem. 

8.  FACILITY  CODE  (FC):  Code  used  to  identify  the 
specific  type  of  facility  that  either  generated  the  lesson 
or  is  affected  by  the  lesson.  Examples  of  facility  types 
include  administrative,  dormitories,  military  family 
housing,  maintenance ,  etc. 

9.  WORK  TYPE:  Used  to  identify  what  type  of  work 
generated  or  is  affected  by  the  lesson.  Examples  include 
mechanical,  electrical,  civil,  structural,  communications, 
etc. 


10.  LESSON  TYPE:  Three  choices  —  management, 
technical  or  both.  Osed  to  help  reduce  volume  of  lessons 
searched  according  to  user’s  needs. 

All  of  the  interviewees  related  that  the  final 
selection  of  lesson  content  may  need  to  be  adjusted  after  an 
initial  trial  period.  It  is  anticipated  most  users  will 
find  or  suggest  additional  content  areas  and  the  system 
should  be  flexible  enough  to  adapt  as  required. 

Format ,  The  specific  format  of  a  lesson  learned 
was  not  considered  critical  by  the  interviewees.  The  one 
important  aspect  of  format  that  must  not  be  overlooked 
however,  is  consistency.  All  of  the  lessons  input  into  the 
system,  whether  submitted  by  different  commands,  bases  or 
levels  of  management,  must  be  consistent.  If  the  user  has 
to  waste  time  from  lesson  to  lesson  adjusting  to  new  formats 
with  similar  information  in  different  areas,  the  user  is 
likely  to  become  dissatisfied  with  the  system.  The 
interviewees  agreed  any  format  similar  to  that  of  the  NALL 
or  AFLL  lessons  would  be  sufficient  for  the  lessons. 
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Search/Retrieval.  Input.  Print/Sort.  The  most 
important  aspects  of  the  proposed  lessons-learned  system  are 
lesson  search/ retrieval ,  input  and  print/sort.  Unless  these 
functions  are  easy  to  accomplish,  most  users  will  avoid 
using  the  system.  All  of  the  interviewees  expressed  little 
concern  regarding  how  the  data  was  physically  stored  within 
the  computer  system.  Instead,  the  procedure  the  users  must 
follow  to  search/ retrieve,  input,  and  print/sort  the 
lessons-learned  data  was  considered  very  important. 

Search/Retrieve .  The  SEARCH/RETRIEVE  menu  option 
should  allow  the  user  to  perform  searches  of  the  lessons- 
learned  database  based  on  user  specified  keywords  or  full 
text  phrases,  facility  code,  work  type,  lesson  type  or 
impact  areas  is  necessary.  A  SEARCH/RETRIEVE  menu,  similar 
to  that  shown  in  Figure  19,  should  appear  on  the  screen  when 
the  user  selects  the  SEARCH/RETRIEVE  option  from  the  main 
menu . 

The  user  should  be  allowed  to  Lag  the  type  of  searches 
desired  and  then  be  presented  additional  menus  where  the 
specifics  of  each  type  of  search  can  be  entered.  Sample 
menus  for  keyword,  phrase  and  facility  code  searches  are 
shown  in  Figures  20,  21  and  22  respectively.  Once  the 
SEARCH/RETRIEVE  series  has  been  completed,  the  user  should 
be  returned  to  the  main  menu.  From  the  main  menu  the  user 
can  then  initiate  additional  searches,  input  lessons  or 
print/sort  the  lessons  retrieved. 
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FACILITY  ACQUISITION  LESSONS  LEARNED 
SEARCH /RETRIEVE  MENU 

p  Keyword  Search 
Phrase  Search 

■  Facility  Code  Search 

■  Work  Type  Search 
Lesson  _Type  Search 

i  Impact  Area  Search; 


<Arrows>  to  Move,  Tag  Option(s)  with  <Space>, 
Hit  <Enter>  When  Ready... 
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FF-1  Help 

PF-2  Prt  Sen 

PF-3  Utilities  i  PF-4  Shell 

PF-5  Cancel 

PF-6  Previous 

PF-7  Mam  M.  !  PF-16  Exit 

Figure  19.  Proposed  Search/Retrieve  Menu 
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Select  Lesson  Areas  to  Search: 

TOPIC  ■  LESSON  LEAR  In  ED 

■  PROBLEM  L  ARP R OP ’  AC Ti 6 N 

■  DISCUSSION  ALL  AREAS 


<Arrows>  to  Move,  Tag  Option!  s)  with  <  Space>, 
Hit  <Enter>  Wnen  Ready... 


PF-1  Help 

PF-2  Prt  Sen 

PF-3  Utilities 

PF-4  Shell 

PF-5  Cancel 

|  PF-8  Previous 

PF-7  Mam  M. 

PF-10  Exit 

Figure  20.  Proposed  Keyword  Search  Menu 
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FACILITY  ACQUISITION  LESSONS  LEARNED 
PHRASE  SEARCH 


Enter  Phrase: 


Case  Sensitive?  [ YES  NO 

Select  Lesson  Areas  to  Search: 

TOPIC  LESSON  LEARNED 

PROBLEM  APPROP.  ACTION 

DISCUSSION  ALL  AREAS 

<Arrows>  to  Move,  Tag  Option(s)  with  <Space>, 


Hit  <Enter>  When  Ready... 


PF-1  Help 

PF-2  Prt  Sen 

PF-3  Utilities 

PF-4  Shell 

PF-5  Cancel 

PF-6  Previous: 

PF-7  Main  M. 

PF-1 9  Bltt. 

Figure  21.  Proposed  Phrase  Search  Menu 


FACILITY  ACQUISITION  LESSONS  LEARNED 
FACILITY  CODE  SEARCH 

Select  Type  of  Search  Desired:  [0§  AND 

Enter  Facility  Codes  OR  <Shift-PF8> 
to  Select  irrom  Master  List  of  Codes: 


<Arrows>  to  Move, 

Hit  <Enter>  When  Ready... 


■3SE5H 

PF-2  Prt  Sen 

PF-3  Utilities 

PF-4  Shell 

PF-5  Cancel 

PF-8  Previous! 

PF-7  Main  M. 

PF-1 6  Exit 

Figure  22.  Proposed  Facility  Code  Search  Menu 


96 


Input  Lessons.  Engineers,  inspectors,  project 
manag--  a,  designers  and  construction  managers  should  be 
allowed  access  to  input  lessons.  Physical  input  of  lessons 
should  be  accomplished  using  a  fill  in  the  blank  approach. 
Selecting  the  INPOT  LESSONS  option  at  the  main  menu  should 
present  the  user  with  a  blank  form  showing  each  of  the  major 
areas  listed  in  the  content  section  of  this  chapter.  A 
sample  blank  form  is  shown  in  Figure  23. 

LESSON  NUMBER:  xxxx  TOPIC : 

IMPACT  AREAS:  _ 

LESSON  LEARNED:  _ 


PROBLEM: 


DISCUSSION: 


APPROP.  ACTION: 


FACILITY  WORK  LESSON 

CODE : _ TYPE : _ TYPE : _ 

(up  to  6)  (up  to  3)  (select  1) 

Facility  code,  work  type  and  lesson  type  master 
indexes  may  be  obtained  by  pressing  PF-6.  Select 
up  to  the  number  indicated. 

Figure  23.  Proposed  Lesson  Input  Form 
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The  system  should  allow  the  user  to  fill  in  each  of  the 
blank  areas,  edit  areas  previously  filled  in  and  save  the 
completed  lesson.  Lesson  sections  such  as  impact  areas, 
facility  codes,  work  type  and  lesson  type  should  allow  the 
user  to  retrieve  master  indexes  of  the  approved  areas , codes 
and  types  and  select  from  the  master  indexes  by  toggling  a 
light  bar  or  hitting  a  "select  this"  key.  When  the  user  has 
completed  filling  in  the  form,  the  system  should  add  a 
lesson  learned  number  and  allow  the  user  to  save,  edit  or 
totally  discard  the  lesson.  Selection  of  one  of  these 
functions  should  then  return  the  user  to  the  main  menu. 
Lessons -learned  should  be  validated  by  HQ  AFESC. 

Print/Sort .  The  PRINT/SORT  capabilities  of  the 
system  should  allow  the  user  to  sort  the  lessons  on  facility 
code,  work  type,  lesson  type  and  impact  area  --  or  any 
combination  of  these.  Using  a  PRINT/SORT  Menu  similar  to 
that  shown  in  Figure  24,  the  user  should  be  able  to  tag  or 
select  the  desired  fields  for  sorting.  From  this  menu  the 
user  should  also  be  provided  a  means  to  select  the 
destination  for  the  retrieved  and  sorted  lessons.  Potential 
destinations  include  the  screen,  printer  or  file.  Once  the 
user  has  made  the  desired  selections  and  hit  <Enter> ,  the 
system  should  prompt  the  user,  "Do  you  want  an  Index  and 
Table  of  Contents  generated?".  Next,  the  system  should  sort 
the  lessons,  and  send  them  to  the  desired  destination.  If 
the  destination  is  a  file,  tl*e  user  should  be  allowed  to 
specify  the  filename,  extension,  directory,  etc. 
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FACILITY  ACQUISITION  LESSONS  LEARNED 
PRINT /SORT  MENU 

Select  Sort  Methods)  and  Destination  for  Lessons: 

■  Facility  Code  Screen 

■  Work  Type  ■  Printer 

I^ssonType  File 

[  Impact"  Areal 

<Arroirs>  to  Move,  Taf  Option(s)  with  <Space>, 
Hit  <Knter>  When  Ready... 


PF-1  Help 

PF-2  Prt  Sen 

PF-3  Utilities 

PF-4  Shell 

PF-5  Cancel 

PF-1 6  Exit 

Figure  24.  Proposed  Print /Sort  Menu 


The  PRINT/SORT  Menu,  shown  in  Figure  24,  indicates  the 
user  has  selected  to  sort  the  retrieved  lessons  on  facility 
code  and  work  type.  The  destination  for  the  lessons  is  the 
printer.  When  the  user  hits  the  <Enter>  key  the  system  will 
then  prompt  for  the  filename,  extension,  directory,  etc. 

Figures  depicting  the  work  type,  lesson  type  and  impact 
area  search  menus  are  not  shown  here.  These  menus  would  be 
similar  to  the  facility  code  search  menu  shown  in  Figure  22. 
All  of  the  menus  shown  in  this  chapter  are  only  suggested 
menus  developed  through  the  interviews  with  the  users.  All 
of  the  interviewees  agreed  that  the  proposed  menus  shown 
here  are  adequate  to  initiate  a  prototype  system.  It  is 
expected  changes  in  the  menus  will  be  necessary  after  the 
users  have  used  and  become  familiar  the  system. 
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Summary 

This  chapter  presented  the  findings  regarding  the 
research  question,  "From  the  user’s  perspective,  how  should 
the  proposed  lessons-learned  MIS  be  structured?''. 

Interviews  with  personnel  from  the  AFRCEs  and  HQ  AFLC/DER 
established  a  baseline  description  of  the  proposed  MIS.  The 
interviewees  all  stated  the  proposed  system  should  be  WANG- 
based,  because  the  WANG  is  common  to  all  AFRCEs  and  Civil 
Engineering  units. 

An  "ideal  system"  was  related  by  the  users  which  would 
allow  capturing  and  retrieving  lessons-learned  on  both  a 
visual  and  audio  basis.  From  this  ideal  system,  the 
interviewees  supplied  several  insights  into  what  a  realistic 
system  would  be,  within  the  financial  and  technical 
capabilities  of  Civil  Engineering. 

The  proposed  lessons-learned  MIS  should  be  menu-driven 
with  minimal  requirements  for  the  user  to  type  in  a 
response.  Instead,  the  user  should  be  afforded  the 
capability  of  selecting  from  a  preset  menu  with  very  few 
keystrokes.  Each  level  of  the  menus  should  be  easily 
accessible  and  it  should  be  equally  easy  to  retrace  or  "back 
out"  of  any  level.  Programmed  function  keys  should  be  used 
only  for  those  functions  common  to  all  menus  such  as  help, 
cancel,  shell,  exit,  etc.  Lightbar  selection  of  menu 
specific  options,  such  as  SEARCH/RETRIEVE,  INPOT  LESSONS  and 
PRINT/SORT,  should  be  available  to  the  user. 
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Lessons- learned  should  be  generated  and  input  into  the 
system  by  those  most  familiar  wi on  Civil  Engineering 
facility  acquisition.  In  particular,  engineers,  designers, 
construction  managers,  project  managers  and  inspectors 
should  have  full  access  to  the  proposed  system.  The 
principle  system  should  be  located  at  HQ/AFESC,  Tyndall  AFB. 

Lessons  should  be  standardized  in  format,  but  not 
constrained  by  limits  in  length  or  information  content. 
Lessons-learned  content  should  include  --  impact  areas, 
topic,  lesson-learned,  problem,  discussion,  appropriate 
action,  facility  codes,  work  type  and  lesson  type.  If 
possible,  lessons  should  contain  both  sanitized  information 
that  does  not  reveal  who  actually  was  involved  with  the  case 
from  which  the  lesson  was  learned,  and  unsanitized 
information  with  very  specific  points  of  contact.  Non-Air 
Force  Civil  Engineering  users,  such  as  the  COE,  NAVFAC  and 
contractors,  should  be  limited  to  the  sanitized  portion  of 
the  lessons . 

Chapter  VI  presents  the  summary,  conclusions  and 
recommendations  of  this  research. 
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The  objective  of  this  research  was  to  identify  factors 
and  procedures  that  should  be  considered  when  developing  a 
construction  management  oriented,  lessons-leamed  management 


information  system  for  the  Civil  Engineering  WANG  computer. 

To  meet  that  objective  this  study  considered  several 
questions,  including: 

1 .  How  are  generic  management  information  systems 
developed? 

2.  Are  there  on-line  "lessons- learned"  management 
information  systems  in  use?  If  so,  how  have  these 
systems  been  developed? 

3.  What  common  factors  made  the  existing  systems 
successful  or  less  than  successful? 

4.  How  should  a  WANG-based  lessons-learned  system  be 
developed  to  meet  the  needs  of  AF  construction 
managers? 

5 .  From  the  user ’ s  perspective ,  how  should  the  WANG 
lessons-learned  MIS  be  structured? 

A  review  of  the  literature  revealed  management 
information  systems  development  has  progressed  from  the 
traditional  systems  analysis  methodology  to  the  more  recent, 
user-developed  methodology . 

The  traditional  systems  analysis  methodology  consisted 
of  seven  activities:  1)  preliminary  investigation,  2) 
requirements  analysis,  3)  prototype  development,  4)  system 
design,  5)  software  development,  6)  testing  and  7) 
implementation.  The  traditional  systems  analysis 
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methodology  was  often  a  very  lengthy  process  because  the 
systems  analyst  had  to  first  determine  exactly  what  the  user 
needed.  As  a  result,  the  traditional  approach  frequently 
failed  to  meet  the  user’s  needs  both  in  terms  of  time  and 
system  function.  The  more  recent  user-developed  methodology 
—  using  prototypes  and  fourth  generation  application 
generators  —  put  the  user  in  more  direct  control  of  the 
systems  development.  The  systems  analyst  must  still  provide 
the  structure  and  discipline  required  in  systems 
development . 

An  understanding  of  WANG-specif ic  applications 
development  methodologies  was  obtained  from  the  systems 
personnel  at  HQ  AFESC.  Due  to  the  high  demand  for  WANG 
applications  in  Civil  Engineering,  most  WANG  specific 
applications  are  user-developed  through  functional 
applications  workshops.  Functional  applications  workshops 
bring  together  the  functional  users  of  a  proposed  system; 
here,  they  determine  the  data  content,  menu  structure  and 
systems-data  interface  requirements.  Once  the  user 
requirements  are  defined,  a  prototype  is  built.  The 
prototype  then  becomes  a  working  model  for  the  functional 
users  to  evaluate,  refine  and  improve.  Iteratively,  a  new 
application  becomes  a  reality. 

Several  on-line  management  information  systems  were 
explored,  both  general  and  lessons-learned  oriented.  Two  of 
the  largest  DOD  lessons-learned  MIS’s  —  the  Air  Force 
Lessons-Leamed  System,  and  the  Naval  Aviations 
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Lessons-Learned  System  --  were  researched  in  terms  of 
development  methodology,  content  and  performance.  Both  the 
AFLL  and  the  NALL  were  developed  using  a  combination  of  the 
traditional  systems  analysis  and  user-developed  prototype 
methodologies .  Both  the  AFLL  and  NALL  contain  short 
descriptive  lessons -learned  dealing  with  weapons  systems 
acquisitions.  Finally,  both  systems  performance  appears  to 
be  meeting  the  needs  of  the  users  quite  well. 

Several  factors  contribute  to  the  success  of  management 
information  systems .  The  most  important  factors  are  system 
capability,  user-friendliness,  data  accuracy  and  user 
involvement  in  the  systems  development  stage.  The  user- 
developed  approach  to  systems  development  increases  all  of 
these  factors. 

From  the  Civil  Engineering  construction  managers 
perspective,  a  facilities  acquisition  lessons- learned  system 
should  be  WANG-based,  menu-driven,  user-friendly  and  easily 
adaptable  to  the  changing  needs  of  the  user.  User 
recommendations  for  the  characteristics  of  the  system,  data 
and  manipulation  of  the  data  in  the  lessons-learned  system 
were  obtained  through  interviews  with  the  AFRCEs. 

According  to  the  users ,  the  proposed  system  should  make 
maximum  use  of  menus.  From  the  menus  the  user  should  be 
able  to  select  options  with  minimal  keystrokes .  Other  than 
the  input  of  lessons  learned,  the  users  wished  to  avoid 
having  to  fill  out  lengthy  blank  forms,  as  is  currently 
required  with  many  of  the  WANG/WIMS  report  generators. 
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Menus  should  allow  option  selection  by  lightbars.  Only 
those  functions  that  are  common  to  all  menus,  such  as  print 
screen,  previous  screen,  cancel,  main  menu,  and  exit  should 
be  PF-key  based. 

All  levels  of  Air  Force  Civil  Engineering  construction 
should  have  access  to  the  system  for  input,  search  and 
retrieval.  Limited  access  to  sanitized  lessons  (free  of 
references  to  specific  people,  organizations  and  programs) 
should  be  available  to  contractors  and  other  DoD 
construction  personnel. 

The  system  should  be  WANG-based,  and  centrally  located 
at  HQ  AFESC.  Lesson  validation  should  be  performed  by  HQ 
AFESC.  Because  the  CPO  costs  associated  with  operating  the 
WANG  minicomputer  are  not  significant  compared  to  mainframe 
computer  CPU  costs,  the  lessons  learned  system  need  not  be 
supplemented  with  a  bulletin  board  type  system  similar  to 
that  used  by  the  Navy.  The  menus  envisioned  by  the  users 
are  illustrated  in  Chapter  V. 

Conclusions 

This  study  began  with  a  search  for  the  for  the 
definitive  specifications  of  a  WANG  based,  lessons-learned, 
management  information  system  --  for  Civil  Engineering 
construction  managers.  However,  just  as  there  is  never  one 
best  way  to  do  almost  anything,  there  are  no  unequivocal 
specifications  for  the  proposed  lessons-learned  MIS. 

Instead  of  quantifying  and  freezing  the  requirements 
for  the  proposed  lessons-learned  system,  this  research 
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determined  the  best  requirements  for  lessons- learned  systems 
are  open  and  flexible.  This  research  verified  the  need  for 
a  lessons-learned  system  to  overcome  the  inexperience  facing 
many  USAF  construction  managers. 

A  starting  point,  or  descriptive  prototype  for  the 
proposed  system  is  available  as  a  result  of  this  research. 
Using  the  menus  depicted  in  Chapter  V,  and  the  narrative 
explanations  of  the  interactions  and  functions  of  those 
menus,  a  working  prototype  could  quickly  be  generated  using 
fourth  generation  application  generators.  Once  this 
prototype  is  provided  to  the  construction  managers,  further 
recommendations  for  system  improvements  are  sure  to  follow. 

Recommendations  for  Fuirther  Research 

Considerable  time  and  effort  in  this  research  was 
expended  determining  exactly  what  a  lessons-learned 
management  information  systems  is,  how  it  works  and  if  it 
works.  Likewise,  finding,  using  and  assessing  existing 
systems  consumed  much  of  the  limited  research  time 
available.  Consequently,  in  hind-sight,  insufficient  time 
was  available  to  determine  specifically  how  a  WANG  based, 
lessons-learned  system  should  be  structured. 

Travel  time  and  funding  limitations  prevented  gathering 


opinions  and  ideas  from  all  levels  of  Air  Force  Civil 
Engineering  construction  management.  Instead,  only  AFRCE 
level  managers  insights  were  obtained. 


Additional  research  should  be  conducted  to  further  the 


development  of  a  lessons-learned  HIS  for  construction 
managers.  The  findings  presented  in  Chapter  V  should  be 
converted  into  an  actual  prototype  by  HQ  AFESC,  Tyndall  AFB. 
That  prototype  should  then  be  provided  to  as  many  Civil 
Engineering  construction  management  areas  as  possible. 

After  sufficient  time  has  been  allowed  the  users  to 
evaluate  the  prototype,  additional  recommendations  and 
concepts  should  be  collected  either  through  surveys  or 
delphi  groups  Iteratively,  a  working  MIS  could  and  should 
be  developed.  Although  there  may  be  no  substitute  for 
experience  —  using  lessons-learned  is  as  close  as  one  can 
get. 

Additional  research  could  also  be  conducted  to 
determine  if  and  how  the  existing  Air  Force  Lessons  Learned 
system  at  HQ  AFALD/LSL  could  be  used  to  meet  the  needs  of 
Air  Force  Civil  Engineering  construction  managers.  This 
research  focused  on  the  development  of  a  WANG-based  lessons 
learned  system,  but  the  Air  Force  Lessons  Learned  system  is 
also  a  viable  alternative  that  should  be  explored. 
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Appendix  A: 


[This  first  series  of  questions  is  designed  to  provide  a 
simple  profile  of  you,  the  user.  Please  feel  free  to 
elaborate  on  any  area  addressed  or  provide  information 
beyond  what  is  asked  if  you  consider  it  important.  Above 
all,  there  are  no  right  or  wrong  answers.  Again,  the 
purpose  of  this  portion  of  the  interview  is  to  provide  a 
profile  of  you,  the  user]. 


USER  PROFILE: 


1 .  Name : 

2. 

Date : 

3.  Current  Position: 

4. 

Phone : 

5.  Bow  many  years  experience  do  you  have  in  Const.  Mgt? 

6.  What  scope  of  construction  have  you  worked? 

7.  What  roles  or  positions  have  you  held  or  performed? 

8.  Are  you  familiar  with  computers?  To  what  extent? 

9.  Which  ones  are  you  most  familiar  with? 

10 .  Have  you  ever  written  computer  programs? 

11.  What  languages? 

12.  Do  you  use  computer(s)  in  your  current  job? 

13.  For  what  purposes? 

14.  Are  you  familiar  with  the  WANG/WIMS? 

15.  How  familiar?  (Have  you  used  it?  For  what  purposes?) 

16.  Do  you  find  the  WANG  user-friendly?  (why  or  why  not) 

17.  Are  their  other  system  you  find  more  user-friendly? 

18.  Which  ones  and  why? 

19.  Are  you  familiar  with  the  WANG  conventions?  (PF  keys, 
etc ) 

20.  Are  you  familiar  with  the  concept  of  on-line  lessons 
learned  systems?  Have  you  ever  used  one? 
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[This  second  series  of  questions  is  designed  to  allow  you  to 
provide  your  input  on  the  proposed  system.  The  questions 
are  grouped  into  three  areas:  general  system  parameters, 
lessons-learned  structure  and  lessons-leamed  input  and 
retrieval.  In  simple  terms,  these  three  areas  address  the 
system,  data  and  data  handling.  Again,  there  are  no  right 
or  wrong  answers.  Feel  free  to  take  any  time  required  to 
think  about  an  answer.  If  you  prefer,  we  can  pass  and 
return  to  any  question(s)  you  require  more  time  to  think 
about.  If  necessary  I  can  even  call  you  back  later  -- 
today,  tomorrow  or  next  week.  Please  consider  yourself 
constrained  only  by  your  imagination.] 


GENERAL  SYSTEM  PARAMETERS: 

21.  If  you  could  "design"  a  LL  system  what  would  it  be  like? 

22.  What  system  would  it  be  on? 

23.  What  type  of  interface  would  it  use?  (mouse,  icons, 
pull  down  menus,  menu  bars,  command  driven,  etc. ) 

24.  If  you  used  a  menu-driven  system,  how  many  submenus 
would  you  consider  necessary?  (Print,  sort,  search, 
any  others? ) 

25.  How  much  help  facilities  should  be  included? 

26.  Where  should  printing  capability  be  included? 

27.  Should  usage  be  mandatory? 

27.  Who  should  have  access  to  the  system? 

29.  Any  further  ideas  on  the  system  in  general? 


LESSONS-LEARNED  STRUCTURE 

30.  How  long  should  a  typical  lesson  be? 

31.  Is  there  a  maximum  recommended  length? 

32.  What  major  types  of  lessons  would  you  envision? 

33.  What  level  of  data  would  you  include  in  lessons? 

34.  How  would  you  organize  the  data  to  constitute  a  lesson? 
(cat  code,  fac  type,  specification  section,  other?) 
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35 .  Ara  there  any  keywords  or  codes  that  each  lesson  should 
contain  such  as  facility  type,  facility  area,  etc? 

36.  Should  the  lessons  be  santized? 

37.  Any  additional  ideas  on  the  structure /content  of  the 
lessons? 


LESSONS -LEARNED  INPUT/T’ETRIEVAL 

38.  In  general,  how  would  the  lessons  be  stored/retrieved? 

39.  Who  should  write  the  lessons? 

40 .  How  should  the  lessons  be  added  to  the  system? 

41.  Who  should  have  capability  of  adding  or  deleting 
lessons? 

42.  How  should  the  lessons  be  retrievable? 

43.  Is  word  searching  or  full  phrase  searching  necessary? 

44.  If  so,  how  many  keywords  are  necessary? 

45.  Is  indexing  or  generating  a  report  important? 

46.  Should  system  indicate  "X  lessons  found?” 

47.  After  search  is  complete,  how  would  you  want  the  data 
portrayed?  (Lesson  titles,  abstracts,  leasson  learned, 
discussion,  recommended  action,  all  or  some  of  above?) 

48.  Is  lessons  sorting  capability  necessary? 

49.  Any  additional  ideas  on  lessons  input/retrieval? 


[Your  assistance  in  this  research  of  on-line  lessons-learned 
systems  is  greatly  appreciately .  Your  time  and  insight  into 
thi3  study  has  helped  to  ensure  that  when  a  facilities 
acquisition  lessons-learned  system  is  built,  it  will  meet 
the  needs  of  the  user.  If  you  think  of  any  additional 
information  you  might  like  to  add  later,  feel  free  to  call 
me.  Again,  thank  you.] 
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